Jul 14 18:54:14.357996 ip-10-0-132-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 18:54:14.744346 ip-10-0-132-123 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 18:54:14.744346 ip-10-0-132-123 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 18:54:14.744346 ip-10-0-132-123 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 18:54:14.744346 ip-10-0-132-123 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 18:54:14.744346 ip-10-0-132-123 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 18:54:14.745460 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.745372    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 18:54:14.751718 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751690    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 18:54:14.751718 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751715    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 18:54:14.751718 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751719    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 18:54:14.751718 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751724    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751727    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751731    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751734    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751736    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751739    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751743    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751746    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751748    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751750    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751753    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751755    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751758    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751761    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751764    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751766    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751769    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751771    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751774    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751777    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 18:54:14.751898 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751779    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751784    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751789    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751792    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751795    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751798    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751801    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751805    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751813    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751816    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751819    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751822    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751825    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751828    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751830    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751832    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751835    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751839    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751842    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 18:54:14.752376 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751844    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751847    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751850    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751853    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751855    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751858    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751860    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751864    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751866    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751869    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751871    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751874    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751877    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751880    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751882    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751885    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751887    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751890    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751892    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751895    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 18:54:14.752858 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751897    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751901    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751903    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751906    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751909    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751912    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751914    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751917    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751919    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751922    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751924    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751927    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751929    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751932    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751934    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751937    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751940    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751942    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751947    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751949    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 18:54:14.753384 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751952    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751956    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751958    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751961    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751964    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751966    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751968    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751971    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751974    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751976    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751979    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751981    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751983    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751988    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751991    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751994    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.751997    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753449    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753458    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 18:54:14.753908 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753461    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753464    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753467    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753470    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753472    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753475    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753477    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753480    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753482    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753485    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753487    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753490    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753493    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753496    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753498    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753502    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753504    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753507    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753509    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753512    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 18:54:14.754406 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753514    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753517    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753521    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753526    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753529    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753531    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753534    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753536    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753539    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753541    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753543    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753546    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753548    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753551    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753554    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753556    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753558    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753561    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753563    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753566    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 18:54:14.754933 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753568    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753570    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753573    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753575    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753578    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753580    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753582    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753586    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753590    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753592    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753595    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753597    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753599    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753602    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753604    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753607    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753609    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753612    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753614    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753631    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 18:54:14.755459 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753634    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753637    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753639    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753642    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753644    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753647    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753649    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753653    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753655    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753658    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753660    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753663    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753665    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753669    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753673    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753676    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753678    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753681    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753684    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753687    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 18:54:14.756038 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753689    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753693    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753695    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753698    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753702    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753709    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753712    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753715    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753718    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753722    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753724    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753727    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753730    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753732    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753734    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753737    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.753757    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754179    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 18:54:14.756543 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754185    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754188    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754191    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754194    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754196    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754199    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754201    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754204    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754206    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754209    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754211    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754213    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754216    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754219    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754221    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754225    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754227    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754230    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754233    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 18:54:14.757022 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754238    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754241    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754244    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754246    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754249    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754252    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754254    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754257    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754259    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754262    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754264    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754266    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754269    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754271    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754273    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754276    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754278    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754281    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754283    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754285    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 18:54:14.757503 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754288    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754290    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754293    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754296    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754298    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754300    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754303    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754306    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754308    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754311    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754314    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754316    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754319    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754321    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754324    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754326    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754329    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754331    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754334    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754336    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 18:54:14.758014 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754338    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754341    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754343    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754346    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754348    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754351    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754353    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754355    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754358    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754360    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754364    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754367    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754370    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754373    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754375    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754378    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754381    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754383    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754386    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 18:54:14.758519 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754389    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754391    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754394    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754396    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754399    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754401    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754404    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754406    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754410    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754413    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754416    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754419    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754422    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754424    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754427    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754429    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754432    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754435    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754437    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 18:54:14.759012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.754441    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754515    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754523    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754530    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754536    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754543    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754548    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754560    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754564    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754567    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754570    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754573    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754577    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754581    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754584    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754587    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754590    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754593    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754596    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754600    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754603    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754606    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754609    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754612    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 18:54:14.759472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754615    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754634    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754637    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754640    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754643    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754646    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754649    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754651    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754655    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754659    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754662    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754665    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754667    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754670    2562 flags.go:64] FLAG: --enable-server="true"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754673    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754678    2562 flags.go:64] FLAG: --event-burst="100"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754681    2562 flags.go:64] FLAG: --event-qps="50"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754683    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754686    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754688    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754695    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754698    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754700    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754703    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754706    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 18:54:14.760171 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754709    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754712    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754715    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754718    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754720    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754723    2562 flags.go:64] FLAG: --feature-gates=""
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754726    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754729    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754732    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754734    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754737    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754740    2562 flags.go:64] FLAG: --help="false"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754743    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-123.ec2.internal"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754745    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754748    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754751    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754754    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754757    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754760    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754763    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754765    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754768    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754770    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754773    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 18:54:14.760860 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754776    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754779    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754781    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754786    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754789    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754792    2562 flags.go:64] FLAG: --lock-file=""
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754794    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754797    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754799    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754804    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754808    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754810    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754813    2562 flags.go:64] FLAG: --logging-format="text"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754816    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754819    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754822    2562 flags.go:64] FLAG: --manifest-url=""
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754824    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754828    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754831    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754835    2562 flags.go:64] FLAG: --max-pods="110"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754838    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754841    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754843    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754846    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754849    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 18:54:14.761480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754852    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754855    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754863    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754866    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754869    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754872    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754875    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754880    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754883    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754886    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754888    2562 flags.go:64] FLAG: --port="10250"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754893    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754896    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2626647a757979b"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754899    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754902    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754904    2562 flags.go:64] FLAG: --register-node="true"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754907    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754911    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754915    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754918    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754920    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754924    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754927    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754929    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754932    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 18:54:14.762128 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754935    2562 flags.go:64] FLAG: --runonce="false"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754938    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754941    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754944    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754947    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754950    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754952    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754955    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754958    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754960    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754963    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754966    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754969    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754971    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754974    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754977    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754982    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754984    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754988    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754992    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754997    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.754999    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755002    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755005    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755008    2562 flags.go:64] FLAG: --v="2"
Jul 14 18:54:14.762772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755012    2562 flags.go:64] FLAG: --version="false"
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755017    2562 flags.go:64] FLAG: --vmodule=""
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755020    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.755024    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755147    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755151    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755154    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755157    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755160    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755163    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755165    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755167    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755170    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755172    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755176    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755178    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755180    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755183    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755185    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755188    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755190    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 18:54:14.763422 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755193    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755195    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755197    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755200    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755203    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755206    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755212    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755215    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755217    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755220    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755223    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755225    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755229    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755236    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755239    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755242    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755244    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755247    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755249    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 18:54:14.763964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755252    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755254    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755257    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755259    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755262    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755264    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755266    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755269    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755272    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755275    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755280    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755283    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755286    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755289    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755292    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755295    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755297    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755301    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755304    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 18:54:14.764436 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755308    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755310    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755313    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755316    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755319    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755322    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755324    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755327    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755329    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755332    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755334    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755337    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755339    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755342    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755344    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755347    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755349    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755352    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755354    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755356    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 18:54:14.764953 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755359    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755361    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755363    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755366    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755368    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755371    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755373    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755376    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755378    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755381    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755384    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755387    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755391    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755393    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755396    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755398    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755400    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755403    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755405    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755408    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 18:54:14.765434 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755410    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 18:54:14.765971 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755413    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 18:54:14.765971 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755416    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 18:54:14.765971 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.755418    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 18:54:14.765971 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.756009    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 18:54:14.767311 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.767287    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 18:54:14.767311 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.767311    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 18:54:14.767368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.767331    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 18:54:14.767368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.767337    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767395    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767399    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767403    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767406    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767410    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767412    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767415    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767417    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767420    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767422    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767425    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767428    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767430    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 18:54:14.767428 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767433    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767435    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767438    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767440    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767443    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767446    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767448    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767450    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767453    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767457    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767461    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767463    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767467    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767469    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767472    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767474    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767478    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767481    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767483    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 18:54:14.767795 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767486    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767488    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767490    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767493    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767495    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767497    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767500    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767502    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767504    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767507    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767509    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767511    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767514    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767516    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767519    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767522    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767525    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767527    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767529    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767532    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 18:54:14.768254 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767534    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767536    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767539    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767541    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767544    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767546    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767549    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767551    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767553    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767556    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767559    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767562    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767564    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767567    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767569    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767572    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767575    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767577    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767580    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767582    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 18:54:14.768764 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767585    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767589    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767594    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767596    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767599    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767602    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767604    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767607    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767610    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767612    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767615    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767631    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767634    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767637    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767640    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767643    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767645    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767648    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767651    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767653    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 18:54:14.769288 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767655    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767658    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767660    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767663    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767666    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767668    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767672    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.767678    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767799    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767804    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767807    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767809    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767812    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767815    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767818    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 18:54:14.769793 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767821    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767824    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767827    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767829    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767832    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767835    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767837    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767840    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767842    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767845    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767847    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767849    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767852    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767854    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767857    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767859    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767861    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767864    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767866    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767869    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 18:54:14.770189 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767871    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767874    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767876    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767878    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767881    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767883    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767885    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767888    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767890    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767892    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767895    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767897    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767900    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767902    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767905    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767909    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767913    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767916    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767919    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767922    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 18:54:14.770768 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767925    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767927    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767930    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767932    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767935    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767938    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767941    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767944    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767946    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767949    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767951    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767953    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767955    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767958    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767961    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767963    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767965    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767968    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767970    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 18:54:14.771264 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767972    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767975    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767977    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767980    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767982    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767985    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767988    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767990    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767992    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767995    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.767998    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768000    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768002    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768004    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768007    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768009    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768012    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768014    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768016    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768019    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 18:54:14.771838 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768021    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768025    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768028    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768031    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768033    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768036    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768038    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768041    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768044    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768046    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768048    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768051    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:14.768053    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.768058    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 18:54:14.772331 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.768185    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 18:54:14.772693 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.770189    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 18:54:14.772693 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.771420    2562 server.go:1019] "Starting client certificate rotation"
Jul 14 18:54:14.772693 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.771526    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 18:54:14.772693 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.771564    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 18:54:14.774120 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.774105    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 18:54:14.776908 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.776892    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 18:54:14.785612 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.785594    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 14 18:54:14.787559 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.787506    2562 log.go:25] "Validated CRI v1 image API"
Jul 14 18:54:14.789731 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.789712    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 18:54:14.792299 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.792279    2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bfe93479-fb85-448a-98f0-4f0417393d8d:/dev/nvme0n1p3 f7f19643-da18-4059-9efb-ba640753d3f5:/dev/nvme0n1p4]
Jul 14 18:54:14.792370 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.792299    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 18:54:14.798039 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.797923    2562 manager.go:216] Machine: {Timestamp:2026-07-14 18:54:14.796388153 +0000 UTC m=+0.324727651 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101065 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eb1bddfe35eeb81262c30303f4b77 SystemUUID:ec2eb1bd-dfe3-5eeb-8126-2c30303f4b77 BootID:058ef632-472e-4aa6-94da-7eeefff63acc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:72:7e:cc:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:72:7e:cc:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:16:f5:11:67:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 18:54:14.798039 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.798033    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 18:54:14.798177 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.798165    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 18:54:14.801561 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.801528    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 18:54:14.801734 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.801565    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 18:54:14.802231 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.802219    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 18:54:14.802231 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.802232    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 18:54:14.802302 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.802239    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 18:54:14.802949 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.802938    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 18:54:14.803952 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.803934    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 18:54:14.804159 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.804146    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 18:54:14.807110 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807098    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 18:54:14.807244 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807234    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 18:54:14.807276 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807264    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 18:54:14.807303 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807291    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 18:54:14.807341 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807310    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 18:54:14.807341 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807324    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 14 18:54:14.807341 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807334    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 18:54:14.807741 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807729    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 18:54:14.807789 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.807750    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 18:54:14.810077 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.810059    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 18:54:14.811907 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.811892    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 18:54:14.811973 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.811924    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 18:54:14.812077 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812066    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 18:54:14.812107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812086    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 18:54:14.812107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812094    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 18:54:14.812107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812099    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 18:54:14.812107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812108    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 18:54:14.812209 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812114    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 18:54:14.812602 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812593    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 18:54:14.812602 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812603    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 18:54:14.812680 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812611    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 18:54:14.812680 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812631    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 18:54:14.812680 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812641    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 18:54:14.812680 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.812650    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 18:54:14.814126 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.814113    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 18:54:14.814126 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.814124    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 18:54:14.817242 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.817223    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 18:54:14.817303 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.817223    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 18:54:14.817303 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.817249    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 18:54:14.817372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.817325    2562 server.go:1268] "Started kubelet"
Jul 14 18:54:14.817516 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.817425    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 18:54:14.817516 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.817505    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 18:54:14.817686 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.817642    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 18:54:14.818377 ip-10-0-132-123 systemd[1]: Started Kubernetes Kubelet.
Jul 14 18:54:14.821514 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.821490    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 18:54:14.822425 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.822382    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 18:54:14.822553 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.822536    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 18:54:14.827486 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.827471    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 18:54:14.827572 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.827533    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 18:54:14.827643 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.827583    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:14.828417 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828398    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 18:54:14.828507 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828420    2562 factory.go:55] Registering systemd factory
Jul 14 18:54:14.828507 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828429    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 14 18:54:14.828600 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828552    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 18:54:14.828600 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828562    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 18:54:14.828600 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828574    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828652    2562 factory.go:153] Registering CRI-O factory
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828667    2562 factory.go:223] Registration of the crio container factory successfully
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828675    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828683    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828701    2562 factory.go:103] Registering Raw factory
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.828716    2562 manager.go:1196] Started watching for new ooms in manager
Jul 14 18:54:14.828756 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.827519    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-123.ec2.internal.18c23ccd4b0dd765  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-123.ec2.internal,UID:ip-10-0-132-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-123.ec2.internal,},FirstTimestamp:2026-07-14 18:54:14.817298277 +0000 UTC m=+0.345637782,LastTimestamp:2026-07-14 18:54:14.817298277 +0000 UTC m=+0.345637782,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-123.ec2.internal,}"
Jul 14 18:54:14.829565 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.829544    2562 manager.go:349] Starting recovery of all containers
Jul 14 18:54:14.834136 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.834114    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dj565"
Jul 14 18:54:14.834557 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.834529    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 18:54:14.834740 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.834715    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 18:54:14.837314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.837267    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 18:54:14.839725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.839710    2562 manager.go:354] Recovery completed
Jul 14 18:54:14.841237 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.841215    2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 14 18:54:14.842135 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.842117    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dj565"
Jul 14 18:54:14.844370 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.844355    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:14.846186 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.846171    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:14.846271 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.846205    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:14.846271 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.846222    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:14.846909 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.846893    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:14.847392 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.847379    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 18:54:14.847392 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.847392    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 18:54:14.847456 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.847407    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 18:54:14.847990 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.847968    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:14.848090 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.848001    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:14.848090 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.848012    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:14.848163 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.848054    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-123.ec2.internal.18c23ccd4cc6a81c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-123.ec2.internal,UID:ip-10-0-132-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-123.ec2.internal,},FirstTimestamp:2026-07-14 18:54:14.846187548 +0000 UTC m=+0.374527051,LastTimestamp:2026-07-14 18:54:14.846187548 +0000 UTC m=+0.374527051,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-123.ec2.internal,}"
Jul 14 18:54:14.848504 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.848491    2562 policy_none.go:49] "None policy: Start"
Jul 14 18:54:14.848541 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.848509    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 18:54:14.848541 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.848520    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 18:54:14.849581 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.849571    2562 policy_none.go:47] "Start"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896252    2562 manager.go:337] "Starting Device Plugin manager"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.896440    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896451    2562 server.go:87] "Starting device plugin registration server"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896692    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896702    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896783    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896859    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.896868    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.897374    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 18:54:14.899091 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.897428    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:14.959165 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.959134    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 18:54:14.959165 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.959179    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 18:54:14.959385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.959203    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 18:54:14.959385 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:14.959236    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 18:54:14.961688 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.961668    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 18:54:14.997737 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.997718    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:14.998723 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.998704    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:14.998811 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.998737    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:14.998811 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.998748    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:14.998811 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:14.998772    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.009950 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.009931    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.010002 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.009962    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-123.ec2.internal\": node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.042037 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.042009    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.059310 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.059282    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"]
Jul 14 18:54:15.059418 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.059370    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:15.061106 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.061086    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:15.061214 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.061124    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:15.061214 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.061147    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:15.062422 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.062405    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:15.062555 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.062533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.062613 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.062563    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:15.063197 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.063180    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:15.063264 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.063212    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:15.063264 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.063222    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:15.063264 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.063185    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:15.063362 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.063277    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:15.063362 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.063293    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:15.064583 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.064569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.064686 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.064598    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 18:54:15.065345 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.065325    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 18:54:15.065424 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.065349    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 18:54:15.065424 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.065363    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeHasSufficientPID"
Jul 14 18:54:15.090338 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.090313    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-123.ec2.internal\" not found" node="ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.093860 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.093842    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-123.ec2.internal\" not found" node="ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.130014 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.129984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee475eee2cc4e4319a99b8043bc5a81e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal\" (UID: \"ee475eee2cc4e4319a99b8043bc5a81e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.130014 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.130016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3d36c070043dd6db6d4a5194d07329-config\") pod \"kube-apiserver-proxy-ip-10-0-132-123.ec2.internal\" (UID: \"ad3d36c070043dd6db6d4a5194d07329\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.130162 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.130032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee475eee2cc4e4319a99b8043bc5a81e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal\" (UID: \"ee475eee2cc4e4319a99b8043bc5a81e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.143103 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.143080    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.230239 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.230196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3d36c070043dd6db6d4a5194d07329-config\") pod \"kube-apiserver-proxy-ip-10-0-132-123.ec2.internal\" (UID: \"ad3d36c070043dd6db6d4a5194d07329\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.230239 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.230234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee475eee2cc4e4319a99b8043bc5a81e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal\" (UID: \"ee475eee2cc4e4319a99b8043bc5a81e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.230239 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.230249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee475eee2cc4e4319a99b8043bc5a81e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal\" (UID: \"ee475eee2cc4e4319a99b8043bc5a81e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.230464 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.230301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee475eee2cc4e4319a99b8043bc5a81e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal\" (UID: \"ee475eee2cc4e4319a99b8043bc5a81e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.230464 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.230306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3d36c070043dd6db6d4a5194d07329-config\") pod \"kube-apiserver-proxy-ip-10-0-132-123.ec2.internal\" (UID: \"ad3d36c070043dd6db6d4a5194d07329\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.230464 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.230344    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee475eee2cc4e4319a99b8043bc5a81e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal\" (UID: \"ee475eee2cc4e4319a99b8043bc5a81e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.243329 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.243300    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.344267 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.344194    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.392413 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.392376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.397239 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.397222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:15.444478 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.444447    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.544993 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.544948    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.645575 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.645492    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.746020 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.745981    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.771161 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.771132    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 18:54:15.771336 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.771315    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 18:54:15.845184 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.845148    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:49:16 +0000 UTC" deadline="2028-04-13 07:50:20.138114972 +0000 UTC"
Jul 14 18:54:15.845184 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.845179    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15324h56m4.292939849s"
Jul 14 18:54:15.846249 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.846225    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.886541 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:15.886491    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee475eee2cc4e4319a99b8043bc5a81e.slice/crio-53eab751e03aa306c39b3b7b24a38d2ae29d03a2dd5db89998e04d1dfbdf6f59 WatchSource:0}: Error finding container 53eab751e03aa306c39b3b7b24a38d2ae29d03a2dd5db89998e04d1dfbdf6f59: Status 404 returned error can't find the container with id 53eab751e03aa306c39b3b7b24a38d2ae29d03a2dd5db89998e04d1dfbdf6f59
Jul 14 18:54:15.891148 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.891133    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 18:54:15.947200 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:15.947119    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:15.962353 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:15.962311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal" event={"ID":"ee475eee2cc4e4319a99b8043bc5a81e","Type":"ContainerStarted","Data":"53eab751e03aa306c39b3b7b24a38d2ae29d03a2dd5db89998e04d1dfbdf6f59"}
Jul 14 18:54:16.047931 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:16.047897    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:16.075836 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:16.075813    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 18:54:16.103850 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:16.103823    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 18:54:16.148518 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:16.148486    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:17.904782 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:17.904759    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-123.ec2.internal\" not found"
Jul 14 18:54:17.926942 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:17.926913    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3d36c070043dd6db6d4a5194d07329.slice/crio-68b9dfdb11abeca2f0eac50c4e53c7e45c841a3ed447f8789b425263757963df WatchSource:0}: Error finding container 68b9dfdb11abeca2f0eac50c4e53c7e45c841a3ed447f8789b425263757963df: Status 404 returned error can't find the container with id 68b9dfdb11abeca2f0eac50c4e53c7e45c841a3ed447f8789b425263757963df
Jul 14 18:54:17.958021 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:17.957988    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 18:54:17.984285 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:17.984254    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:17.995099 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:17.995064    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 18:54:17.996132 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:17.996111    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal"
Jul 14 18:54:18.007396 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.007376    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 18:54:18.464459 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.464419    2562 apiserver.go:52] "Watching apiserver"
Jul 14 18:54:18.472882 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.472855    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 18:54:18.474050 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.474022    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal","openshift-multus/multus-7qqsb","openshift-multus/multus-additional-cni-plugins-rdqhg","openshift-ovn-kubernetes/ovnkube-node-4zsmh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv","openshift-cluster-node-tuning-operator/tuned-zrxld","openshift-image-registry/node-ca-xl9rj","openshift-multus/network-metrics-daemon-jnqtz","openshift-network-diagnostics/network-check-target-qg9bg","openshift-network-operator/iptables-alerter-f8ds8","kube-system/global-pull-secret-syncer-ch6sn","kube-system/konnectivity-agent-2v9cv","kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal","openshift-dns/node-resolver-rwrp6"]
Jul 14 18:54:18.475798 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.475764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.476997 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.476976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.477097 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.477066    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:18.478189 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.478167    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 18:54:18.478296 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.478278    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.478362 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.478172    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzj6n\""
Jul 14 18:54:18.478362 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.478336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.478494 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.478169    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.479767 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.479734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.480593 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.480569    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 18:54:18.480735 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.480685    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 18:54:18.480944 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.480926    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.481032 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.480944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.481230    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.481236    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfcdn\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.481499    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.482085    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.482260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.482774    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483050    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483233    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-85wks\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483607    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483814    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483888    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483984    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.483993    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-56jqk\""
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.484072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.484134    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:18.484313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.484235    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 18:54:18.485379 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.485182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.486097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.486064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.486612 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.486374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.486612 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.486381    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.486612 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.486426    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f6dwn\""
Jul 14 18:54:18.487248 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.487226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.487888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.487869    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5dwqj\""
Jul 14 18:54:18.488068 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.488052    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.488155 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.488142    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.489037 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.489009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 18:54:18.489972 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.489954    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r9kkj\""
Jul 14 18:54:18.490474 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.490436    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 18:54:18.491378 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.490828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:18.491378 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.490921    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:18.491378 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.491064    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rcmq4\""
Jul 14 18:54:18.491638 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.491617    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 18:54:18.492583 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.491963    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 18:54:18.492583 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.492550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.494040 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-var-lib-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494130 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvt7v\" (UniqueName: \"kubernetes.io/projected/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-kube-api-access-fvt7v\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494130 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494081    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-etc-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-system-cni-dir\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.494230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.494230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-var-lib-kubelet\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-device-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovnkube-script-lib\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsx4h\" (UniqueName: \"kubernetes.io/projected/918653c0-439e-4d9d-81ad-e777db1b9eab-kube-api-access-vsx4h\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-modprobe-d\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-tmp\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-ovn\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhpt\" (UniqueName: \"kubernetes.io/projected/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-kube-api-access-wmhpt\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/139756f3-c948-4ecc-b8e1-a222e73af648-dbus\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-sys\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-run-ovn-kubernetes\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494446 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-host\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-log-socket\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4z2\" (UniqueName: \"kubernetes.io/projected/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-kube-api-access-7v4z2\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-registration-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvc6\" (UniqueName: \"kubernetes.io/projected/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-kube-api-access-lkvc6\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/918653c0-439e-4d9d-81ad-e777db1b9eab-host\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/139756f3-c948-4ecc-b8e1-a222e73af648-kubelet-config\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysctl-conf\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovn-node-metrics-cert\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-sys-fs\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-slash\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.494856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cnibin\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysctl-d\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-systemd\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-cni-netd\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-socket-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-tmp-dir\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.494992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-run-netns\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-node-log\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-env-overrides\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovnkube-config\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495133    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495163    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kw9nc\""
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/918653c0-439e-4d9d-81ad-e777db1b9eab-serviceca\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysconfig\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.495730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-kubernetes\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-kubelet\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-systemd-units\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-hosts-file\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-os-release\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-kube-api-access-hhm7c\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-systemd\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-tuned\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-cni-bin\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495561    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-run\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.496357 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.495584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-lib-modules\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.499669 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.499651    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 18:54:18.524278 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.524253    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmncr"
Jul 14 18:54:18.538569 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.538543    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmncr"
Jul 14 18:54:18.585727 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.585695    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 18:54:18.595983 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.595950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-sys-fs\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.595995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-slash\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-host-slash\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-sys-fs\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cnibin\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysctl-d\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-systemd\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596117    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-slash\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-cni-netd\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596158 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-socket-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-tmp-dir\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.596229    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-run-netns\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.596298    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:19.09627475 +0000 UTC m=+2.968832966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-run-netns\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-node-log\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cnibin\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-env-overrides\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596521    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysctl-d\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e02c79d6-96ac-4231-a203-bcd7c713fa6c-konnectivity-ca\") pod \"konnectivity-agent-2v9cv\" (UID: \"e02c79d6-96ac-4231-a203-bcd7c713fa6c\") " pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-systemd\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab4bf931-2464-4177-be55-46c7ea27d9db-cni-binary-copy\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-cni-netd\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.596705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovnkube-config\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/918653c0-439e-4d9d-81ad-e777db1b9eab-serviceca\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysconfig\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-kubernetes\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-kubelet\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-systemd-units\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-hosts-file\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtjd\" (UniqueName: \"kubernetes.io/projected/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-kube-api-access-nhtjd\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-system-cni-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-os-release\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/d552c277-89f4-4fec-8e29-1cd663363215-kube-api-access-wg7dk\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.596965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-netns\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-hostroot\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-multus-certs\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-kube-api-access-hhm7c\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-systemd\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.597314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-tuned\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-env-overrides\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-cni-bin\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-daemon-config\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-node-log\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-run\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-lib-modules\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-hosts-file\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-var-lib-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-cnibin\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597329    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-etc-kubernetes\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvt7v\" (UniqueName: \"kubernetes.io/projected/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-kube-api-access-fvt7v\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-etc-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-socket-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-cni-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-systemd\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-socket-dir-parent\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-system-cni-dir\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-var-lib-kubelet\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-device-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovnkube-script-lib\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsx4h\" (UniqueName: \"kubernetes.io/projected/918653c0-439e-4d9d-81ad-e777db1b9eab-kube-api-access-vsx4h\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e02c79d6-96ac-4231-a203-bcd7c713fa6c-agent-certs\") pod \"konnectivity-agent-2v9cv\" (UID: \"e02c79d6-96ac-4231-a203-bcd7c713fa6c\") " pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-k8s-cni-cncf-io\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-tmp-dir\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-cni-multus\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-kubelet\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-conf-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-modprobe-d\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-tmp\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.598862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-ovn\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhpt\" (UniqueName: \"kubernetes.io/projected/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-kube-api-access-wmhpt\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/139756f3-c948-4ecc-b8e1-a222e73af648-dbus\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.597980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-os-release\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkh2\" (UniqueName: \"kubernetes.io/projected/ab4bf931-2464-4177-be55-46c7ea27d9db-kube-api-access-jfkh2\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-sys\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-run-ovn-kubernetes\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598132    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-system-cni-dir\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-host\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-log-socket\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4z2\" (UniqueName: \"kubernetes.io/projected/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-kube-api-access-7v4z2\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.599711 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-registration-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598336    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvc6\" (UniqueName: \"kubernetes.io/projected/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-kube-api-access-lkvc6\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/918653c0-439e-4d9d-81ad-e777db1b9eab-host\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/139756f3-c948-4ecc-b8e1-a222e73af648-kubelet-config\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-iptables-alerter-script\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-cni-bin\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysctl-conf\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovn-node-metrics-cert\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cni-binary-copy\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-etc-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-ovn\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-sys\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598656    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-host\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-run-ovn-kubernetes\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-log-socket\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-kubernetes\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.598990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.600594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599036    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-run-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovnkube-config\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-registration-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599151    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-kubelet\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599195    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-systemd-units\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599244    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/918653c0-439e-4d9d-81ad-e777db1b9eab-host\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/918653c0-439e-4d9d-81ad-e777db1b9eab-serviceca\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysconfig\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/139756f3-c948-4ecc-b8e1-a222e73af648-kubelet-config\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/139756f3-c948-4ecc-b8e1-a222e73af648-dbus\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-sysctl-conf\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-lib-modules\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-host-cni-bin\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-run\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-device-dir\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-modprobe-d\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.599911    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-var-lib-openvswitch\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.601452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.600220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-os-release\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.602358 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.600339    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-var-lib-kubelet\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.602358 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.600363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovnkube-script-lib\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.602358 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.600854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.603523 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.603498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-ovn-node-metrics-cert\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.604471 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.604368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-etc-tuned\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.604471 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.604444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-tmp\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.607846 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.607825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhm7c\" (UniqueName: \"kubernetes.io/projected/e8b4c8df-84c4-41c0-b7eb-43eed56a6e35-kube-api-access-hhm7c\") pod \"multus-additional-cni-plugins-rdqhg\" (UID: \"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35\") " pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.608475 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.608353    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvt7v\" (UniqueName: \"kubernetes.io/projected/4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe-kube-api-access-fvt7v\") pod \"tuned-zrxld\" (UID: \"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe\") " pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.609326 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.608741    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:18.609326 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.608768    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:18.609326 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.608781    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:18.609326 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.608847    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:19.108829727 +0000 UTC m=+2.981387931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:18.611340 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.611317    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvc6\" (UniqueName: \"kubernetes.io/projected/a3a0c652-ff2b-4ada-8de8-584ef8d6166b-kube-api-access-lkvc6\") pod \"node-resolver-rwrp6\" (UID: \"a3a0c652-ff2b-4ada-8de8-584ef8d6166b\") " pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.613114 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.612974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsx4h\" (UniqueName: \"kubernetes.io/projected/918653c0-439e-4d9d-81ad-e777db1b9eab-kube-api-access-vsx4h\") pod \"node-ca-xl9rj\" (UID: \"918653c0-439e-4d9d-81ad-e777db1b9eab\") " pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.613114 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.613025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4z2\" (UniqueName: \"kubernetes.io/projected/ec7f6d5f-fbd4-463a-a7dd-6f1d66870478-kube-api-access-7v4z2\") pod \"ovnkube-node-4zsmh\" (UID: \"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478\") " pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.616722 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.616698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhpt\" (UniqueName: \"kubernetes.io/projected/c3e0f0c6-4dda-4465-94f2-94efc76a0b28-kube-api-access-wmhpt\") pod \"aws-ebs-csi-driver-node-zdtzv\" (UID: \"c3e0f0c6-4dda-4465-94f2-94efc76a0b28\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.624115 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.624073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal" event={"ID":"ad3d36c070043dd6db6d4a5194d07329","Type":"ContainerStarted","Data":"68b9dfdb11abeca2f0eac50c4e53c7e45c841a3ed447f8789b425263757963df"}
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-cnibin\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-etc-kubernetes\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-cni-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-socket-dir-parent\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e02c79d6-96ac-4231-a203-bcd7c713fa6c-agent-certs\") pod \"konnectivity-agent-2v9cv\" (UID: \"e02c79d6-96ac-4231-a203-bcd7c713fa6c\") " pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.699991    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-k8s-cni-cncf-io\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-cni-multus\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-kubelet\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-conf-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-os-release\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkh2\" (UniqueName: \"kubernetes.io/projected/ab4bf931-2464-4177-be55-46c7ea27d9db-kube-api-access-jfkh2\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-iptables-alerter-script\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-cni-bin\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-host-slash\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e02c79d6-96ac-4231-a203-bcd7c713fa6c-konnectivity-ca\") pod \"konnectivity-agent-2v9cv\" (UID: \"e02c79d6-96ac-4231-a203-bcd7c713fa6c\") " pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab4bf931-2464-4177-be55-46c7ea27d9db-cni-binary-copy\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.702923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtjd\" (UniqueName: \"kubernetes.io/projected/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-kube-api-access-nhtjd\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-system-cni-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/d552c277-89f4-4fec-8e29-1cd663363215-kube-api-access-wg7dk\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-netns\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-hostroot\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-multus-certs\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.700464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-daemon-config\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-daemon-config\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-cnibin\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701196    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-etc-kubernetes\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-cni-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-socket-dir-parent\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.701329    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:18.701385    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:19.201368089 +0000 UTC m=+3.073926288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-host-slash\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701904    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-multus-conf-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-os-release\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.701978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-k8s-cni-cncf-io\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.703803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702449    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-cni-multus\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-iptables-alerter-script\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-cni-bin\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-var-lib-kubelet\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-hostroot\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e02c79d6-96ac-4231-a203-bcd7c713fa6c-konnectivity-ca\") pod \"konnectivity-agent-2v9cv\" (UID: \"e02c79d6-96ac-4231-a203-bcd7c713fa6c\") " pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-netns\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-host-run-multus-certs\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.702738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab4bf931-2464-4177-be55-46c7ea27d9db-system-cni-dir\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.703102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab4bf931-2464-4177-be55-46c7ea27d9db-cni-binary-copy\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.704653 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.704251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e02c79d6-96ac-4231-a203-bcd7c713fa6c-agent-certs\") pod \"konnectivity-agent-2v9cv\" (UID: \"e02c79d6-96ac-4231-a203-bcd7c713fa6c\") " pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.714285 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.714251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtjd\" (UniqueName: \"kubernetes.io/projected/a87cf829-cb57-48c4-a3ab-c82c2a7913f5-kube-api-access-nhtjd\") pod \"iptables-alerter-f8ds8\" (UID: \"a87cf829-cb57-48c4-a3ab-c82c2a7913f5\") " pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.716147 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.716086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkh2\" (UniqueName: \"kubernetes.io/projected/ab4bf931-2464-4177-be55-46c7ea27d9db-kube-api-access-jfkh2\") pod \"multus-7qqsb\" (UID: \"ab4bf931-2464-4177-be55-46c7ea27d9db\") " pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.716516 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.716500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7dk\" (UniqueName: \"kubernetes.io/projected/d552c277-89f4-4fec-8e29-1cd663363215-kube-api-access-wg7dk\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:18.787281 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.787252    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xl9rj"
Jul 14 18:54:18.796453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.796428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rdqhg"
Jul 14 18:54:18.797024 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.796996    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918653c0_439e_4d9d_81ad_e777db1b9eab.slice/crio-81d0c405d038a5577b16acba7e40e9e52480e051e03c5d3a1e4d18f4c917a76d WatchSource:0}: Error finding container 81d0c405d038a5577b16acba7e40e9e52480e051e03c5d3a1e4d18f4c917a76d: Status 404 returned error can't find the container with id 81d0c405d038a5577b16acba7e40e9e52480e051e03c5d3a1e4d18f4c917a76d
Jul 14 18:54:18.804829 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.804804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:18.807307 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.807276    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b4c8df_84c4_41c0_b7eb_43eed56a6e35.slice/crio-0043ce395b6245aff24a5c54ceaf888d57d1cb3b8ed2ab917530ee6e4a29e1da WatchSource:0}: Error finding container 0043ce395b6245aff24a5c54ceaf888d57d1cb3b8ed2ab917530ee6e4a29e1da: Status 404 returned error can't find the container with id 0043ce395b6245aff24a5c54ceaf888d57d1cb3b8ed2ab917530ee6e4a29e1da
Jul 14 18:54:18.809939 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.809917    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv"
Jul 14 18:54:18.816028 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.815973    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7f6d5f_fbd4_463a_a7dd_6f1d66870478.slice/crio-f6835ab156778ece56743c9c27fc9e2f6ca0bac62e0d4c704ccf68c7bf83dca3 WatchSource:0}: Error finding container f6835ab156778ece56743c9c27fc9e2f6ca0bac62e0d4c704ccf68c7bf83dca3: Status 404 returned error can't find the container with id f6835ab156778ece56743c9c27fc9e2f6ca0bac62e0d4c704ccf68c7bf83dca3
Jul 14 18:54:18.816588 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.816568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zrxld"
Jul 14 18:54:18.820486 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.820460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e0f0c6_4dda_4465_94f2_94efc76a0b28.slice/crio-5db87aa0a7760f266f4384204d41b305987674392e1a56133b05e3d402291a92 WatchSource:0}: Error finding container 5db87aa0a7760f266f4384204d41b305987674392e1a56133b05e3d402291a92: Status 404 returned error can't find the container with id 5db87aa0a7760f266f4384204d41b305987674392e1a56133b05e3d402291a92
Jul 14 18:54:18.823323 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.823257    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwrp6"
Jul 14 18:54:18.827012 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.826986    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4009d6ca_f50f_4e1b_ad08_a7fad13d3bbe.slice/crio-e4b1d64b7b4a51b65a0615126f82803b72a98758b818ef16742a6a8f373247cc WatchSource:0}: Error finding container e4b1d64b7b4a51b65a0615126f82803b72a98758b818ef16742a6a8f373247cc: Status 404 returned error can't find the container with id e4b1d64b7b4a51b65a0615126f82803b72a98758b818ef16742a6a8f373247cc
Jul 14 18:54:18.830066 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.830019    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qqsb"
Jul 14 18:54:18.834108 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.833032    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a0c652_ff2b_4ada_8de8_584ef8d6166b.slice/crio-a2ac87ccb1600543e78d8c470d04f80daac18762e456e244bfe1c408169f1c6c WatchSource:0}: Error finding container a2ac87ccb1600543e78d8c470d04f80daac18762e456e244bfe1c408169f1c6c: Status 404 returned error can't find the container with id a2ac87ccb1600543e78d8c470d04f80daac18762e456e244bfe1c408169f1c6c
Jul 14 18:54:18.839497 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.839369    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4bf931_2464_4177_be55_46c7ea27d9db.slice/crio-7ed24871f5e2c5266fad54185638c4247b0c73062277118dde76b24372054cc4 WatchSource:0}: Error finding container 7ed24871f5e2c5266fad54185638c4247b0c73062277118dde76b24372054cc4: Status 404 returned error can't find the container with id 7ed24871f5e2c5266fad54185638c4247b0c73062277118dde76b24372054cc4
Jul 14 18:54:18.841562 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.841539    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8ds8"
Jul 14 18:54:18.848850 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.848827    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:18.850242 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.850212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87cf829_cb57_48c4_a3ab_c82c2a7913f5.slice/crio-949919465e1a139a00c3050f05aa3e7e61b9724c1b6a67868b5a14b59aa2a693 WatchSource:0}: Error finding container 949919465e1a139a00c3050f05aa3e7e61b9724c1b6a67868b5a14b59aa2a693: Status 404 returned error can't find the container with id 949919465e1a139a00c3050f05aa3e7e61b9724c1b6a67868b5a14b59aa2a693
Jul 14 18:54:18.861085 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:54:18.861047    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02c79d6_96ac_4231_a203_bcd7c713fa6c.slice/crio-f3f5979c4e84c365d434282390b0b155e599f8fd6cb7a9d134c7781d3ec29203 WatchSource:0}: Error finding container f3f5979c4e84c365d434282390b0b155e599f8fd6cb7a9d134c7781d3ec29203: Status 404 returned error can't find the container with id f3f5979c4e84c365d434282390b0b155e599f8fd6cb7a9d134c7781d3ec29203
Jul 14 18:54:18.861902 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:18.861880    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 18:54:19.103709 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.103672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:19.103894 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.103781    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:19.103894 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.103845    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:20.103828123 +0000 UTC m=+3.976386519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.204995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.205053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.205213    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.205231    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.205245    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.205306    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:20.205286755 +0000 UTC m=+4.077844979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.205739    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:19.205827 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.205790    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:20.205773436 +0000 UTC m=+4.078331651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:19.540597 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.540552    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:49:18 +0000 UTC" deadline="2028-02-25 11:24:30.788092266 +0000 UTC"
Jul 14 18:54:19.540597 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.540595    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14176h30m11.247502009s"
Jul 14 18:54:19.615244 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.615212    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:19.615431 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.615341    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:19.615784 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.615763    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:19.615896 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:19.615875    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:19.632979 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.632942    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerStarted","Data":"0043ce395b6245aff24a5c54ceaf888d57d1cb3b8ed2ab917530ee6e4a29e1da"}
Jul 14 18:54:19.637009 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.636972    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xl9rj" event={"ID":"918653c0-439e-4d9d-81ad-e777db1b9eab","Type":"ContainerStarted","Data":"81d0c405d038a5577b16acba7e40e9e52480e051e03c5d3a1e4d18f4c917a76d"}
Jul 14 18:54:19.639809 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.639776    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2v9cv" event={"ID":"e02c79d6-96ac-4231-a203-bcd7c713fa6c","Type":"ContainerStarted","Data":"f3f5979c4e84c365d434282390b0b155e599f8fd6cb7a9d134c7781d3ec29203"}
Jul 14 18:54:19.644689 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.644646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qqsb" event={"ID":"ab4bf931-2464-4177-be55-46c7ea27d9db","Type":"ContainerStarted","Data":"7ed24871f5e2c5266fad54185638c4247b0c73062277118dde76b24372054cc4"}
Jul 14 18:54:19.653451 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.653418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwrp6" event={"ID":"a3a0c652-ff2b-4ada-8de8-584ef8d6166b","Type":"ContainerStarted","Data":"a2ac87ccb1600543e78d8c470d04f80daac18762e456e244bfe1c408169f1c6c"}
Jul 14 18:54:19.655520 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.655491    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv" event={"ID":"c3e0f0c6-4dda-4465-94f2-94efc76a0b28","Type":"ContainerStarted","Data":"5db87aa0a7760f266f4384204d41b305987674392e1a56133b05e3d402291a92"}
Jul 14 18:54:19.658208 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.658179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"f6835ab156778ece56743c9c27fc9e2f6ca0bac62e0d4c704ccf68c7bf83dca3"}
Jul 14 18:54:19.661890 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.661839    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee475eee2cc4e4319a99b8043bc5a81e" containerID="25be639c96106c3d9084e3ed420e94edf71a27e00105d7d6e017cb262c015564" exitCode=0
Jul 14 18:54:19.661982 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.661913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal" event={"ID":"ee475eee2cc4e4319a99b8043bc5a81e","Type":"ContainerDied","Data":"25be639c96106c3d9084e3ed420e94edf71a27e00105d7d6e017cb262c015564"}
Jul 14 18:54:19.664838 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.664810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8ds8" event={"ID":"a87cf829-cb57-48c4-a3ab-c82c2a7913f5","Type":"ContainerStarted","Data":"949919465e1a139a00c3050f05aa3e7e61b9724c1b6a67868b5a14b59aa2a693"}
Jul 14 18:54:19.681303 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:19.681263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zrxld" event={"ID":"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe","Type":"ContainerStarted","Data":"e4b1d64b7b4a51b65a0615126f82803b72a98758b818ef16742a6a8f373247cc"}
Jul 14 18:54:20.111596 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:20.111557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:20.111777 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.111762    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:20.111837 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.111818    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:22.111805185 +0000 UTC m=+5.984363383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:20.212218 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:20.212179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:20.212423 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:20.212238    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:20.212423 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.212395    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:20.212555 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.212434    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:20.212555 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.212475    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:20.212555 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.212537    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:22.212517534 +0000 UTC m=+6.085075737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:20.212943 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.212924    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:20.213023 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.212978    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:22.212963739 +0000 UTC m=+6.085521943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:20.541086 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:20.541019    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:49:18 +0000 UTC" deadline="2028-02-13 03:05:44.411728909 +0000 UTC"
Jul 14 18:54:20.541086 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:20.541084    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13880h11m23.870648938s"
Jul 14 18:54:20.616786 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:20.616752    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:20.616943 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:20.616887    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:21.615623 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:21.615567    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:21.615982 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:21.615709    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:21.616083 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:21.616065    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:21.616178 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:21.616142    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:22.130079 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:22.129984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:22.130262 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.130158    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:22.130262 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.130234    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:26.130213681 +0000 UTC m=+10.002771893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:22.230851 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:22.230817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:22.231014 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:22.230874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:22.231075 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.231041    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:22.231075 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.231060    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:22.231075 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.231073    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:22.231222 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.231124    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:26.231107417 +0000 UTC m=+10.103665632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:22.231550 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.231525    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:22.231721 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.231574    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:26.231558413 +0000 UTC m=+10.104116625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:22.617911 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:22.617878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:22.618478 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:22.618000    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:23.615830 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:23.615779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:23.615999 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:23.615925    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:23.616242 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:23.616224    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:23.616318 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:23.616300    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:24.616611 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:24.616573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:24.617047 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:24.616745    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:25.615679 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:25.615593    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:25.615823 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:25.615727    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:25.616140 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:25.616118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:25.616257 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:25.616237    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:26.165777 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:26.165724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:26.166178 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.165909    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:26.166178 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.165971    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:34.165954147 +0000 UTC m=+18.038512350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:26.266630 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:26.266589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:26.266803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:26.266648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:26.266803 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.266796    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:26.266897 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.266813    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:26.266897 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.266826    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:26.266897 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.266877    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:34.266863947 +0000 UTC m=+18.139422161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:26.267316 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.267300    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:26.267370 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.267352    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:34.267337913 +0000 UTC m=+18.139896124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:26.616308 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:26.616277    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:26.616516 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:26.616388    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:27.616390 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:27.615783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:27.616390 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:27.615911    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:27.616390 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:27.616271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:27.616390 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:27.616353    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:28.615364 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:28.615333    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:28.615534 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:28.615467    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:29.615549 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:29.615508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:29.615971 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:29.615527    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:29.615971 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:29.615648    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:29.615971 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:29.615754    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:30.615541 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:30.615507    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:30.616004 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:30.615625    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:31.615970 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:31.615929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:31.616422 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:31.615934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:31.616422 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:31.616051    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:31.616422 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:31.616173    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:32.615983 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:32.615944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:32.616380 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:32.616075    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:33.615553 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:33.615531    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:33.615681 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:33.615617    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:33.615681 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:33.615642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:33.615752 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:33.615696    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:34.220138 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:34.220089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:34.220591 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.220278    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:34.220591 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.220352    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:50.220330427 +0000 UTC m=+34.092888626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:34.320445 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:34.320397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:34.320647 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:34.320506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:34.320647 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.320587    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:34.320647 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.320602    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:34.320647 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.320609    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:34.320647 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.320622    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:34.320887 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.320672    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:50.320651464 +0000 UTC m=+34.193209676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:34.320887 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.320693    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:54:50.320683829 +0000 UTC m=+34.193242028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:34.616383 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:34.616291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:34.616549 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:34.616421    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:35.615872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:35.615832    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:35.616301 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:35.615831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:35.616301 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:35.615985    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:35.616301 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:35.616021    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:36.616505 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:36.616469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:36.616965 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:36.616584    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:37.615746 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:37.615706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:37.615746 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:37.615732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:37.615989 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:37.615845    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:37.615989 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:37.615964    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:38.615775 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.615616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:38.616287 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:38.615859    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:38.724313 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.724248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal" event={"ID":"ee475eee2cc4e4319a99b8043bc5a81e","Type":"ContainerStarted","Data":"91e826ef383d4f65a6d09268abb119eb8dd16d5b7721fce99b4b921a00dd6404"}
Jul 14 18:54:38.725705 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.725682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zrxld" event={"ID":"4009d6ca-f50f-4e1b-ad08-a7fad13d3bbe","Type":"ContainerStarted","Data":"d19deb39365c9a0839ccf4393ee69b8380fa1f44ef1e384e1cc9e3fb55ab590a"}
Jul 14 18:54:38.727061 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.727039    2562 generic.go:356] "Generic (PLEG): container finished" podID="e8b4c8df-84c4-41c0-b7eb-43eed56a6e35" containerID="ca898a9194cc33cdb2d9afa90bbdf20e44ae431628dc5d8b8f721057dd127fa6" exitCode=0
Jul 14 18:54:38.727168 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.727110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerDied","Data":"ca898a9194cc33cdb2d9afa90bbdf20e44ae431628dc5d8b8f721057dd127fa6"}
Jul 14 18:54:38.728728 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.728698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xl9rj" event={"ID":"918653c0-439e-4d9d-81ad-e777db1b9eab","Type":"ContainerStarted","Data":"dce9b31314f6d67fe0a745899ee2ddd7759b626dbbc7cd9640270a0eca0ed07a"}
Jul 14 18:54:38.730875 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.730856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal" event={"ID":"ad3d36c070043dd6db6d4a5194d07329","Type":"ContainerStarted","Data":"25f6b559dbcf1128fa7b0b3dc64d93065f8ba6f359ebad82be65f4fe253773d3"}
Jul 14 18:54:38.732225 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.732199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2v9cv" event={"ID":"e02c79d6-96ac-4231-a203-bcd7c713fa6c","Type":"ContainerStarted","Data":"a8ce7455ca94592bdbafe6663f3eedb30666b651c323ba5dbcf9a2f5cca1c22e"}
Jul 14 18:54:38.733599 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.733581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qqsb" event={"ID":"ab4bf931-2464-4177-be55-46c7ea27d9db","Type":"ContainerStarted","Data":"45275daa4c4c80a01a50c0317ab87ba7f22cff87ad28e674553d8ae07d0d3acb"}
Jul 14 18:54:38.734783 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.734768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwrp6" event={"ID":"a3a0c652-ff2b-4ada-8de8-584ef8d6166b","Type":"ContainerStarted","Data":"34c241620ada826ab66fc41803915eee4bcda4ce10dffa789a53c4f4ae292bc0"}
Jul 14 18:54:38.736417 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.736379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv" event={"ID":"c3e0f0c6-4dda-4465-94f2-94efc76a0b28","Type":"ContainerStarted","Data":"183199f7a1d8e8f2445c38603d91c41bc4019cf11d1a03fcaa2c56b8524ca14c"}
Jul 14 18:54:38.741978 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.741960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"89a7574d7c2d94c7847bccca54d41ba7cbda19b9b2051e932841f3abebf80c9e"}
Jul 14 18:54:38.742082 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.742030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"58ca639101ed9e4957cec84e164920b53d0ef2dd517ae74381093db927de53a1"}
Jul 14 18:54:38.742143 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.742083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"e7bc6a9a244a2ba01034d4e7c073ac1ba220f1d12a69170f69615923b1e37282"}
Jul 14 18:54:38.742143 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.742130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"9e99e8ae57cf6acec4d5d88b2d485a9829533be0113ff65ce87243997effae53"}
Jul 14 18:54:38.742241 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.742179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"f30afb5a309707b3daa482c65b55a3e98cf67e138232a38d3c285ef3c611e896"}
Jul 14 18:54:38.743750 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.743712    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-123.ec2.internal" podStartSLOduration=20.743699841 podStartE2EDuration="20.743699841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.743444442 +0000 UTC m=+22.616002662" watchObservedRunningTime="2026-07-14 18:54:38.743699841 +0000 UTC m=+22.616258062"
Jul 14 18:54:38.814270 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.814195    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2v9cv" podStartSLOduration=3.582608095 podStartE2EDuration="22.814172787s" totalImagesPullingTime="19.231564692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.792038882 +0000 UTC m=+22.664597101" watchObservedRunningTime="2026-07-14 18:54:38.814172787 +0000 UTC m=+22.686731008"
Jul 14 18:54:38.837593 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.837522    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-123.ec2.internal" podStartSLOduration=21.837500037 podStartE2EDuration="21.837500037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.814697809 +0000 UTC m=+22.687256029" watchObservedRunningTime="2026-07-14 18:54:38.837500037 +0000 UTC m=+22.710058311"
Jul 14 18:54:38.838084 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.838048    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zrxld" podStartSLOduration=3.577309005 podStartE2EDuration="22.83803883s" totalImagesPullingTime="19.260729825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.837759749 +0000 UTC m=+22.710317969" watchObservedRunningTime="2026-07-14 18:54:38.83803883 +0000 UTC m=+22.710597050"
Jul 14 18:54:38.883524 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.883475    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rwrp6" podStartSLOduration=3.626595941 podStartE2EDuration="22.883459757s" totalImagesPullingTime="19.256863816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.860239433 +0000 UTC m=+22.732797654" watchObservedRunningTime="2026-07-14 18:54:38.883459757 +0000 UTC m=+22.756017974"
Jul 14 18:54:38.883691 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.883603    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7qqsb" podStartSLOduration=3.567451482 podStartE2EDuration="22.883595704s" totalImagesPullingTime="19.316144222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.882869998 +0000 UTC m=+22.755428219" watchObservedRunningTime="2026-07-14 18:54:38.883595704 +0000 UTC m=+22.756153923"
Jul 14 18:54:38.901021 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:38.900966    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xl9rj" podStartSLOduration=8.091658918 podStartE2EDuration="22.900943613s" totalImagesPullingTime="14.809284695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:38.900692385 +0000 UTC m=+22.773250606" watchObservedRunningTime="2026-07-14 18:54:38.900943613 +0000 UTC m=+22.773501833"
Jul 14 18:54:39.027902 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.027813    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:39.165640 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.165604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:39.456316 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.456287    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:39.557035 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.556978    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:39.616197 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.616171    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:39.616197 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.616205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:39.616645 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:39.616282    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:39.616645 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:39.616370    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:39.637946 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.637921    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 18:54:39.748226 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.748187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"2aaf5d871b297e2a7f69b2a434ad76cbfd921d108b47238872b01f6a02ec9d6a"}
Jul 14 18:54:39.749786 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.749756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8ds8" event={"ID":"a87cf829-cb57-48c4-a3ab-c82c2a7913f5","Type":"ContainerStarted","Data":"93075cfc725571a2de3d6dbae544e12af919c80360bf1e84baf5027309910bf6"}
Jul 14 18:54:39.752301 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.752274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv" event={"ID":"c3e0f0c6-4dda-4465-94f2-94efc76a0b28","Type":"ContainerStarted","Data":"56a38f2bf8e6f01a8993ab1be9f9fe80553ad88e824afb2e4a167a9d74871728"}
Jul 14 18:54:39.768018 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.767961    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f8ds8" podStartSLOduration=4.524222856 podStartE2EDuration="23.767942453s" totalImagesPullingTime="19.243719597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:39.76744342 +0000 UTC m=+23.640001640" watchObservedRunningTime="2026-07-14 18:54:39.767942453 +0000 UTC m=+23.640500677"
Jul 14 18:54:39.855947 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.855915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:39.957226 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:39.957190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:40.307233 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.307196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:40.479565 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.479538    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:40.480186 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.480161    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:40.566954 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.566802    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:54:39.637940024Z","UUID":"39292d0c-af24-4c32-aab1-054ff43006bb","Handler":null,"Name":"","Endpoint":""}
Jul 14 18:54:40.568647 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.568621    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 18:54:40.568795 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.568670    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 18:54:40.615352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.615325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:40.615540 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:40.615471    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:40.662876 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.662830    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:40.753881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.753849    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:40.754522 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.754501    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2v9cv"
Jul 14 18:54:40.823173 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:40.823095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:41.107462 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:41.107363    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:41.164814 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:41.164780    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:41.615546 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:41.615356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:41.615706 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:41.615396    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:41.615706 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:41.615641    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:41.615772 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:41.615714    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:41.759905 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:41.759869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"695d975499a3199717d90601b823c58efb40ea64947b2c113ac171d5a38cfe39"}
Jul 14 18:54:42.615298 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:42.615264    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:42.615508 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:42.615429    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:43.616373 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.616183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:43.617032 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.616251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:43.617032 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:43.616476    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:43.617032 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:43.616530    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:43.700727 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.700696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:43.769226 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.769175    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:43.769801 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.769772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" event={"ID":"ec7f6d5f-fbd4-463a-a7dd-6f1d66870478","Type":"ContainerStarted","Data":"d4915624278b383654b73a21e3cd901d646a1f6a7ab073337a65058f8f5d15a4"}
Jul 14 18:54:43.770052 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.769999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:43.771733 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.771700    2562 generic.go:356] "Generic (PLEG): container finished" podID="e8b4c8df-84c4-41c0-b7eb-43eed56a6e35" containerID="ca2281ea187975ab875b6b1dfdc1d93409d711836bf5292351761ee82231a030" exitCode=0
Jul 14 18:54:43.771822 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.771749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerDied","Data":"ca2281ea187975ab875b6b1dfdc1d93409d711836bf5292351761ee82231a030"}
Jul 14 18:54:43.773977 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.773955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv" event={"ID":"c3e0f0c6-4dda-4465-94f2-94efc76a0b28","Type":"ContainerStarted","Data":"1e800d72d2c3cba300361e907137af319f29ec7c130e0995b747360d6325b20a"}
Jul 14 18:54:43.787960 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.787931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:43.807837 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.807794    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh" podStartSLOduration=8.467562002 podStartE2EDuration="27.807779662s" totalImagesPullingTime="19.34021766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:43.806000945 +0000 UTC m=+27.678559165" watchObservedRunningTime="2026-07-14 18:54:43.807779662 +0000 UTC m=+27.680337883"
Jul 14 18:54:43.825379 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:43.825331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdtzv" podStartSLOduration=4.069421341 podStartE2EDuration="27.825316557s" totalImagesPullingTime="23.755895216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:43.82502977 +0000 UTC m=+27.697587990" watchObservedRunningTime="2026-07-14 18:54:43.825316557 +0000 UTC m=+27.697874778"
Jul 14 18:54:44.070268 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.070234    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:44.171141 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.171104    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:44.238981 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.238946    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:44.299829 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.299800    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:44.615774 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.615732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:44.615985 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:44.615850    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:44.777269 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.777217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:44.777807 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.777363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:44.792820 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.792793    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:54:44.978351 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:44.978319    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.052549 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.052508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.245248 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.245172    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.310915 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.310885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.362246 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.362205    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ch6sn"]
Jul 14 18:54:45.362548 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.362342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:45.362548 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:45.362476    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:45.362948 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.362931    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qg9bg"]
Jul 14 18:54:45.363029 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.363019    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:45.363107 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:45.363090    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:45.375970 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.375947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jnqtz"]
Jul 14 18:54:45.376118 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.376049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:45.376175 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:45.376136    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:45.644997 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.644953    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.744673 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.744643    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.781439 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.781335    2562 generic.go:356] "Generic (PLEG): container finished" podID="e8b4c8df-84c4-41c0-b7eb-43eed56a6e35" containerID="a5c903866c3199e605bef91ace321b5c620e9b7544b3e33af74b6fb5e6f5a4e5" exitCode=0
Jul 14 18:54:45.781899 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.781436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerDied","Data":"a5c903866c3199e605bef91ace321b5c620e9b7544b3e33af74b6fb5e6f5a4e5"}
Jul 14 18:54:45.854232 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.854190    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:45.959249 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:45.959214    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:46.620893 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:46.620677    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:46.621057 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:46.620950    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:46.621110 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:46.620736    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:46.621184 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:46.621163    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:47.141211 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.141185    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwrp6_a3a0c652-ff2b-4ada-8de8-584ef8d6166b/dns-node-resolver/0.log"
Jul 14 18:54:47.487677 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.487644    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:47.558142 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.558116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:47.615502 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.615421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:47.615637 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:47.615538    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:47.788225 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.788197    2562 generic.go:356] "Generic (PLEG): container finished" podID="e8b4c8df-84c4-41c0-b7eb-43eed56a6e35" containerID="d00db469dbc168427c7d659cb974d8441ef7648c88848732769ab227cb3dbee2" exitCode=0
Jul 14 18:54:47.788415 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.788254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerDied","Data":"d00db469dbc168427c7d659cb974d8441ef7648c88848732769ab227cb3dbee2"}
Jul 14 18:54:47.941336 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:47.941257    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xl9rj_918653c0-439e-4d9d-81ad-e777db1b9eab/node-ca/0.log"
Jul 14 18:54:48.615656 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:48.615619    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:48.616268 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:48.615780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:48.616268 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:48.615791    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:48.616268 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:48.615861    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:48.687472 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:48.687436    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.065785 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.065523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.173024 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.172977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.274572 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.274483    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.418828 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.418784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.488874 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.488836    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.615912 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.615824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:49.616348 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:49.615931    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:49.733010 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.732975    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:49.834557 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:49.834515    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:50.245531 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:50.245485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:50.245711 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.245637    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:50.245772 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.245732    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret podName:139756f3-c948-4ecc-b8e1-a222e73af648 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:22.245709923 +0000 UTC m=+66.118268146 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret") pod "global-pull-secret-syncer-ch6sn" (UID: "139756f3-c948-4ecc-b8e1-a222e73af648") : object "kube-system"/"original-pull-secret" not registered
Jul 14 18:54:50.346263 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:50.346226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:50.346263 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:50.346269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:50.346541 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.346388    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 18:54:50.346541 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.346388    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:50.346541 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.346421    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 18:54:50.346541 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.346438    2562 projected.go:196] Error preparing data for projected volume kube-api-access-7c2wl for pod openshift-network-diagnostics/network-check-target-qg9bg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:50.346541 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.346481    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl podName:8b167de1-abb3-4ce0-944b-b1731d194365 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:22.346468442 +0000 UTC m=+66.219026640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c2wl" (UniqueName: "kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl") pod "network-check-target-qg9bg" (UID: "8b167de1-abb3-4ce0-944b-b1731d194365") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 18:54:50.346541 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.346496    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs podName:d552c277-89f4-4fec-8e29-1cd663363215 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:22.346490122 +0000 UTC m=+66.219048319 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs") pod "network-metrics-daemon-jnqtz" (UID: "d552c277-89f4-4fec-8e29-1cd663363215") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 18:54:50.616109 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:50.616023    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:50.616109 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:50.616023    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:50.616633 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.616164    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:50.616633 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:50.616302    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:50.999154 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:50.999114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:51.214891 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:51.214837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:51.404211 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:51.404120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:51.615150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:51.615111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:51.615367 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:51.615225    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:51.615367 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:51.615247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:51.860775 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:51.860743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:52.465217 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.465194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:52.618996 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.618917    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:52.619125 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:52.619032    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:52.619125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.619082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:52.619208 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:52.619159    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:52.803236 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.803202    2562 generic.go:356] "Generic (PLEG): container finished" podID="e8b4c8df-84c4-41c0-b7eb-43eed56a6e35" containerID="2b61c9859d356db2258d783e73ac6793b75efa500dafc6204480dc8e274c4606" exitCode=0
Jul 14 18:54:52.803428 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.803253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerDied","Data":"2b61c9859d356db2258d783e73ac6793b75efa500dafc6204480dc8e274c4606"}
Jul 14 18:54:52.821797 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.821766    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:52.926848 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:52.926819    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:53.145901 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.145820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:53.242587 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.242546    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:53.389136 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.389099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:53.589130 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.589095    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:53.615882 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.615841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:53.616027 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:53.615952    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:53.808668 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.808624    2562 generic.go:356] "Generic (PLEG): container finished" podID="e8b4c8df-84c4-41c0-b7eb-43eed56a6e35" containerID="583ba72f328fc7fb5581933b6c53c17a19b9e437dee8083f9f68b5a533cfcad1" exitCode=0
Jul 14 18:54:53.808813 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.808674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerDied","Data":"583ba72f328fc7fb5581933b6c53c17a19b9e437dee8083f9f68b5a533cfcad1"}
Jul 14 18:54:53.994873 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:53.994700    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:54.386179 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:54.386146    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:54.618717 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:54.618628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:54.618717 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:54.618631    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:54.618717 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:54.618724    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:54.618717 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:54.618792    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:54.814131 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:54.814104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" event={"ID":"e8b4c8df-84c4-41c0-b7eb-43eed56a6e35","Type":"ContainerStarted","Data":"bcc2252e19826414d88e5bbf88c41e6d2f00bb9336ae831acffe0c9e3e1c713c"}
Jul 14 18:54:54.860258 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:54.860194    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rdqhg" podStartSLOduration=12.147268459 podStartE2EDuration="38.860179597s" totalImagesPullingTime="26.712911138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:54:54.859936069 +0000 UTC m=+38.732494289" watchObservedRunningTime="2026-07-14 18:54:54.860179597 +0000 UTC m=+38.732737816"
Jul 14 18:54:55.001840 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:55.001784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:55.396879 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:55.396846    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:55.487583 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:55.487555    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:55.615726 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:55.615683    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:55.615922 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:55.615814    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:56.436103 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:56.436070    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:56.534326 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:56.534293    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:56.620235 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:56.620210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:56.620393 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:56.620294    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:56.620393 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:56.620346    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:56.620491 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:56.620442    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:56.653562 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:56.653530    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:57.019076 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:57.019036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:57.190782 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:57.190748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:57.615735 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:57.615697    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:57.616113 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:57.615799    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:57.806500 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:57.806465    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:58.387846 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:58.387817    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:58.615609 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:58.615583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:54:58.615746 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:58.615706    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:54:58.615811 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:58.615779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:54:58.616055 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:58.615870    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:54:58.778519 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:58.778492    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:59.615260 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:59.615213    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:54:59.615466 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:54:59.615336    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:54:59.898013 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:59.897925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:54:59.999005 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:54:59.998966    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:00.156314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:00.156233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:00.256240 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:00.256206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:00.357468 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:00.357439    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:00.459299 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:00.459194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:00.615590 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:00.615554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:00.615782 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:00.615655    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:00.615782 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:00.615711    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:00.615897 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:00.615831    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:01.531200 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:01.531154    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:01.615903 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:01.615870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:01.616119 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:01.615981    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:55:01.632963 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:01.632931    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:02.170981 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:02.170947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:02.229542 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:02.229498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:02.615537 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:02.615500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:02.615938 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:02.615550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:02.615938 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:02.615627    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:02.615938 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:02.615729    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:03.615434 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:03.615385    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:03.615592 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:03.615505    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:55:04.615990 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:04.615960    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:04.616371 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:04.616052    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:04.616371 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:04.616105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:04.616371 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:04.616190    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:04.735447 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:04.735383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:04.836935 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:04.836899    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:05.615686 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:05.615643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:05.615887 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:05.615812    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:55:06.615916 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:06.615882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:06.616437 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:06.615957    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:06.616437 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:06.616012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:06.616437 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:06.616131    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:07.615670 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:07.615634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:07.615840 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:07.615743    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:55:08.616069 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:08.616020    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:08.616531 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:08.616132    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:08.616531 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:08.616186    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:08.616531 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:08.616277    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:09.292242 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:09.292209    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:09.402917 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:09.402885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:09.615924 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:09.615844    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:09.616059 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:09.615950    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:55:10.615362 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:10.615321    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:10.615747 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:10.615332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:10.615747 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:10.615461    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:10.615747 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:10.615504    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:11.615179 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:11.615148    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:11.615344 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:11.615245    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ch6sn" podUID="139756f3-c948-4ecc-b8e1-a222e73af648"
Jul 14 18:55:12.492064 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.492014    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:12.615820 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.615772    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:12.616010 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.615796    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:12.616010 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:12.615889    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qg9bg" podUID="8b167de1-abb3-4ce0-944b-b1731d194365"
Jul 14 18:55:12.616010 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:12.615961    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jnqtz" podUID="d552c277-89f4-4fec-8e29-1cd663363215"
Jul 14 18:55:12.656245 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.656212    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:12.751623 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.751535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:12.848595 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.848560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:12.924347 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.924317    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-123.ec2.internal" event="NodeReady"
Jul 14 18:55:12.924550 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.924503    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 18:55:12.995448 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:12.995387    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78b49755bc-gtvpd"]
Jul 14 18:55:13.018751 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.018673    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-96rqs"]
Jul 14 18:55:13.018904 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.018800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.021561 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.021538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 18:55:13.021675 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.021577    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s87bq\""
Jul 14 18:55:13.021869 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.021846    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 18:55:13.022178 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.022162    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 18:55:13.027643 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.027623    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 18:55:13.042783 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.042741    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b49755bc-gtvpd"]
Jul 14 18:55:13.042783 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.042779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-96rqs"]
Jul 14 18:55:13.042957 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.042878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.046309 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.046285    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 18:55:13.046454 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.046292    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 18:55:13.046454 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.046386    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 18:55:13.046571 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.046456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 18:55:13.046628 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.046598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p26hg\""
Jul 14 18:55:13.103415 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-data-volume\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33f1b5cf-6215-461b-a248-5707fd96cb1d-ca-trust-extracted\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-registry-tls\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77n2v\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-kube-api-access-77n2v\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33f1b5cf-6215-461b-a248-5707fd96cb1d-installation-pull-secrets\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33f1b5cf-6215-461b-a248-5707fd96cb1d-trusted-ca\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-bound-sa-token\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103929 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs565\" (UniqueName: \"kubernetes.io/projected/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-kube-api-access-rs565\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.103929 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33f1b5cf-6215-461b-a248-5707fd96cb1d-registry-certificates\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103929 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33f1b5cf-6215-461b-a248-5707fd96cb1d-image-registry-private-configuration\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.103929 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.103929 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.103818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-crio-socket\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.106074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.106052    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-96h9g"]
Jul 14 18:55:13.124668 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.124625    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jv45g"]
Jul 14 18:55:13.124829 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.124758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.127775 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.127757    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p5svp\""
Jul 14 18:55:13.128312 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.128295    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 18:55:13.128396 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.128303    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 18:55:13.150615 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.150588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-96h9g"]
Jul 14 18:55:13.150747 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.150625    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jv45g"]
Jul 14 18:55:13.150747 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.150678    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.153765 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.153748    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 18:55:13.153872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.153855    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x2zbd\""
Jul 14 18:55:13.154131 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.154120    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 18:55:13.154554 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.154541    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 18:55:13.204430 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4877449-bd14-4f21-ae2c-4337ae659d5a-tmp-dir\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.204430 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf-cert\") pod \"ingress-canary-jv45g\" (UID: \"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf\") " pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.204632 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs565\" (UniqueName: \"kubernetes.io/projected/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-kube-api-access-rs565\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.204632 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4877449-bd14-4f21-ae2c-4337ae659d5a-metrics-tls\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.204632 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxvj\" (UniqueName: \"kubernetes.io/projected/8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf-kube-api-access-6jxvj\") pod \"ingress-canary-jv45g\" (UID: \"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf\") " pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.204632 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33f1b5cf-6215-461b-a248-5707fd96cb1d-registry-certificates\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.204632 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33f1b5cf-6215-461b-a248-5707fd96cb1d-image-registry-private-configuration\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.204632 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204642    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7gcw\" (UniqueName: \"kubernetes.io/projected/e4877449-bd14-4f21-ae2c-4337ae659d5a-kube-api-access-x7gcw\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-crio-socket\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-data-volume\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33f1b5cf-6215-461b-a248-5707fd96cb1d-ca-trust-extracted\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4877449-bd14-4f21-ae2c-4337ae659d5a-config-volume\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-registry-tls\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-crio-socket\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77n2v\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-kube-api-access-77n2v\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.204881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.205316 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33f1b5cf-6215-461b-a248-5707fd96cb1d-installation-pull-secrets\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.205316 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33f1b5cf-6215-461b-a248-5707fd96cb1d-trusted-ca\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.205316 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.204933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-bound-sa-token\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.205316 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.205055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33f1b5cf-6215-461b-a248-5707fd96cb1d-ca-trust-extracted\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.205316 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.205161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-data-volume\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.205583 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.205545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33f1b5cf-6215-461b-a248-5707fd96cb1d-registry-certificates\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.205728 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.205709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.206059 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.206038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33f1b5cf-6215-461b-a248-5707fd96cb1d-trusted-ca\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.209030 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.209005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.209030 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.209021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-registry-tls\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.209153 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.209039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33f1b5cf-6215-461b-a248-5707fd96cb1d-installation-pull-secrets\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.209153 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.209044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33f1b5cf-6215-461b-a248-5707fd96cb1d-image-registry-private-configuration\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.215088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.215067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs565\" (UniqueName: \"kubernetes.io/projected/f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9-kube-api-access-rs565\") pod \"insights-runtime-extractor-96rqs\" (UID: \"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9\") " pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.215729 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.215710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77n2v\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-kube-api-access-77n2v\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.217283 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.217266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33f1b5cf-6215-461b-a248-5707fd96cb1d-bound-sa-token\") pod \"image-registry-78b49755bc-gtvpd\" (UID: \"33f1b5cf-6215-461b-a248-5707fd96cb1d\") " pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.257847 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.257812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:13.306236 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4877449-bd14-4f21-ae2c-4337ae659d5a-config-volume\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.306236 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4877449-bd14-4f21-ae2c-4337ae659d5a-tmp-dir\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.306477 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf-cert\") pod \"ingress-canary-jv45g\" (UID: \"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf\") " pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.306477 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306373    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4877449-bd14-4f21-ae2c-4337ae659d5a-metrics-tls\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.306477 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxvj\" (UniqueName: \"kubernetes.io/projected/8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf-kube-api-access-6jxvj\") pod \"ingress-canary-jv45g\" (UID: \"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf\") " pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.306628 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7gcw\" (UniqueName: \"kubernetes.io/projected/e4877449-bd14-4f21-ae2c-4337ae659d5a-kube-api-access-x7gcw\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.306628 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4877449-bd14-4f21-ae2c-4337ae659d5a-tmp-dir\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.306872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.306841    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4877449-bd14-4f21-ae2c-4337ae659d5a-config-volume\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.308751 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.308730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4877449-bd14-4f21-ae2c-4337ae659d5a-metrics-tls\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.308932 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.308915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf-cert\") pod \"ingress-canary-jv45g\" (UID: \"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf\") " pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.315333 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.315309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxvj\" (UniqueName: \"kubernetes.io/projected/8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf-kube-api-access-6jxvj\") pod \"ingress-canary-jv45g\" (UID: \"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf\") " pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.315520 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.315451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7gcw\" (UniqueName: \"kubernetes.io/projected/e4877449-bd14-4f21-ae2c-4337ae659d5a-kube-api-access-x7gcw\") pod \"dns-default-96h9g\" (UID: \"e4877449-bd14-4f21-ae2c-4337ae659d5a\") " pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.328416 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.328370    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.332275 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.332256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:13.352013 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.351989    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-96rqs"
Jul 14 18:55:13.433794 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.433760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:13.459539 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.459504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jv45g"
Jul 14 18:55:13.540007 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.539869    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-96rqs"]
Jul 14 18:55:13.549071 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:13.549034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ce78ff_e3f1_4c2f_9566_3358fdf3fff9.slice/crio-d3991ba40afa9fc94a1d60334102c421bed1d83cb0f7610013e7573709157953 WatchSource:0}: Error finding container d3991ba40afa9fc94a1d60334102c421bed1d83cb0f7610013e7573709157953: Status 404 returned error can't find the container with id d3991ba40afa9fc94a1d60334102c421bed1d83cb0f7610013e7573709157953
Jul 14 18:55:13.557367 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.557233    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b49755bc-gtvpd"]
Jul 14 18:55:13.560079 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:13.560049    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f1b5cf_6215_461b_a248_5707fd96cb1d.slice/crio-6db3d47c253e2988143916514bc89686cef3ff0520eab1a89c2441e29105b003 WatchSource:0}: Error finding container 6db3d47c253e2988143916514bc89686cef3ff0520eab1a89c2441e29105b003: Status 404 returned error can't find the container with id 6db3d47c253e2988143916514bc89686cef3ff0520eab1a89c2441e29105b003
Jul 14 18:55:13.615216 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.615177    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:13.618230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.618208    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 18:55:13.619726 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.619483    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-96h9g"]
Jul 14 18:55:13.630439 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:13.630396    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4877449_bd14_4f21_ae2c_4337ae659d5a.slice/crio-c705102d861555a59d35d2afb629ea71b250fa991177ac136cf81c9763984669 WatchSource:0}: Error finding container c705102d861555a59d35d2afb629ea71b250fa991177ac136cf81c9763984669: Status 404 returned error can't find the container with id c705102d861555a59d35d2afb629ea71b250fa991177ac136cf81c9763984669
Jul 14 18:55:13.633675 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.633581    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jv45g"]
Jul 14 18:55:13.636998 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:13.636964    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd5555d_a0a4_40a2_ad7d_83273d5fb2bf.slice/crio-edf611184c38356e2bc39ed290d41b19006a6bcab839129578dc08c410ef3305 WatchSource:0}: Error finding container edf611184c38356e2bc39ed290d41b19006a6bcab839129578dc08c410ef3305: Status 404 returned error can't find the container with id edf611184c38356e2bc39ed290d41b19006a6bcab839129578dc08c410ef3305
Jul 14 18:55:13.863241 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.863160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jv45g" event={"ID":"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf","Type":"ContainerStarted","Data":"edf611184c38356e2bc39ed290d41b19006a6bcab839129578dc08c410ef3305"}
Jul 14 18:55:13.864458 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.864432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96rqs" event={"ID":"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9","Type":"ContainerStarted","Data":"d3991ba40afa9fc94a1d60334102c421bed1d83cb0f7610013e7573709157953"}
Jul 14 18:55:13.866024 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.865995    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd" event={"ID":"33f1b5cf-6215-461b-a248-5707fd96cb1d","Type":"ContainerStarted","Data":"9eb87e61baf16f6f8d3d64f21079f1f83e61fae75160a87006e18283a4a9f701"}
Jul 14 18:55:13.866162 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.866045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd" event={"ID":"33f1b5cf-6215-461b-a248-5707fd96cb1d","Type":"ContainerStarted","Data":"6db3d47c253e2988143916514bc89686cef3ff0520eab1a89c2441e29105b003"}
Jul 14 18:55:13.866162 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.866119    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:13.867167 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.867143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-96h9g" event={"ID":"e4877449-bd14-4f21-ae2c-4337ae659d5a","Type":"ContainerStarted","Data":"c705102d861555a59d35d2afb629ea71b250fa991177ac136cf81c9763984669"}
Jul 14 18:55:13.890618 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:13.890556    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd" podStartSLOduration=1.89053257 podStartE2EDuration="1.89053257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:13.888900739 +0000 UTC m=+57.761458959" watchObservedRunningTime="2026-07-14 18:55:13.89053257 +0000 UTC m=+57.763090795"
Jul 14 18:55:14.505224 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.505189    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"]
Jul 14 18:55:14.508432 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.508393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:14.510878 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.510854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 14 18:55:14.510990 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.510912    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q5l7n\""
Jul 14 18:55:14.516818 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.516796    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"]
Jul 14 18:55:14.518852 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.518828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f0190c6-079f-4e91-9018-168e539f5fdf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gqzm8\" (UID: \"1f0190c6-079f-4e91-9018-168e539f5fdf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:14.616272 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.616229    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:14.616749 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.616466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:14.619146 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.619126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f0190c6-079f-4e91-9018-168e539f5fdf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gqzm8\" (UID: \"1f0190c6-079f-4e91-9018-168e539f5fdf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:14.619328 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:14.619258    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 18:55:14.619328 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:14.619321    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0190c6-079f-4e91-9018-168e539f5fdf-tls-certificates podName:1f0190c6-079f-4e91-9018-168e539f5fdf nodeName:}" failed. No retries permitted until 2026-07-14 18:55:15.119303517 +0000 UTC m=+58.991861731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1f0190c6-079f-4e91-9018-168e539f5fdf-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-gqzm8" (UID: "1f0190c6-079f-4e91-9018-168e539f5fdf") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 18:55:14.619575 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.619557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 18:55:14.619655 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.619634    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ltzph\""
Jul 14 18:55:14.619718 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.619696    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 18:55:14.619771 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.619696    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zn8zd\""
Jul 14 18:55:14.619771 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:14.619764    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 18:55:15.123525 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.123482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f0190c6-079f-4e91-9018-168e539f5fdf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gqzm8\" (UID: \"1f0190c6-079f-4e91-9018-168e539f5fdf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:15.126020 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.126002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1f0190c6-079f-4e91-9018-168e539f5fdf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gqzm8\" (UID: \"1f0190c6-079f-4e91-9018-168e539f5fdf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:15.417778 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.417676    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:15.468729 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.468634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:15.566832 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.566800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"]
Jul 14 18:55:15.570594 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.570540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:15.571801 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:15.571773    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0190c6_079f_4e91_9018_168e539f5fdf.slice/crio-a2640f1bac8a2966aa0cf6885c6dbf6a625e3c2f6b5bc2c658c71b54c0bae530 WatchSource:0}: Error finding container a2640f1bac8a2966aa0cf6885c6dbf6a625e3c2f6b5bc2c658c71b54c0bae530: Status 404 returned error can't find the container with id a2640f1bac8a2966aa0cf6885c6dbf6a625e3c2f6b5bc2c658c71b54c0bae530
Jul 14 18:55:15.874971 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:15.874932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8" event={"ID":"1f0190c6-079f-4e91-9018-168e539f5fdf","Type":"ContainerStarted","Data":"a2640f1bac8a2966aa0cf6885c6dbf6a625e3c2f6b5bc2c658c71b54c0bae530"}
Jul 14 18:55:16.798833 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:16.798803    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4zsmh"
Jul 14 18:55:17.004490 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.004452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:17.099863 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.099764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:17.192737 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.192703    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:17.295024 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.294992    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:17.464746 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.464691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:17.565902 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.565865    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:17.883345 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:17.883260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-96h9g" event={"ID":"e4877449-bd14-4f21-ae2c-4337ae659d5a","Type":"ContainerStarted","Data":"a7a4510b278f54aafb5dc3c719523cd4bb4ed29aab3b68e2c80733249dc3078c"}
Jul 14 18:55:19.035596 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:19.035556    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:19.106631 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:19.106593    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:20.616228 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.616188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:20.892841 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.892765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-96h9g" event={"ID":"e4877449-bd14-4f21-ae2c-4337ae659d5a","Type":"ContainerStarted","Data":"86c4868ccc25f395a625b4950a3cecb5e66786aca8e643b3145ece1507654dd0"}
Jul 14 18:55:20.893003 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.892876    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:20.894465 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.894443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jv45g" event={"ID":"8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf","Type":"ContainerStarted","Data":"9d0d7f95cbcc5e989d3835f33bd54b132a71087e596db2fc7bd8f0ebb4748bcd"}
Jul 14 18:55:20.895994 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.895973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96rqs" event={"ID":"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9","Type":"ContainerStarted","Data":"f5c9c6f79fdad20b82c25e1a88bc87393d508f45ac4a91100b96eead566956ff"}
Jul 14 18:55:20.896068 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.896011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96rqs" event={"ID":"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9","Type":"ContainerStarted","Data":"19421740beed8e62a839a55eda1a9ab1cfbe8d75bffc108ff53489e720e796f5"}
Jul 14 18:55:20.913214 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.913162    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-96h9g" podStartSLOduration=4.732062415 podStartE2EDuration="7.91314279s" totalImagesPullingTime="3.181080375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:20.912559444 +0000 UTC m=+64.785117664" watchObservedRunningTime="2026-07-14 18:55:20.91314279 +0000 UTC m=+64.785701005"
Jul 14 18:55:20.934281 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:20.934232    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jv45g" podStartSLOduration=1.746535063 podStartE2EDuration="7.934215085s" totalImagesPullingTime="6.187680022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:20.932834613 +0000 UTC m=+64.805392833" watchObservedRunningTime="2026-07-14 18:55:20.934215085 +0000 UTC m=+64.806773308"
Jul 14 18:55:21.392379 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:21.392343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:21.465096 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:21.465064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:21.902795 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:21.902751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8" event={"ID":"1f0190c6-079f-4e91-9018-168e539f5fdf","Type":"ContainerStarted","Data":"7062e1a79e9072d9193b97ebc81533b8f49f3c96b74cb87310ca0fa4db04b38b"}
Jul 14 18:55:21.921545 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:21.921482    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8" podStartSLOduration=2.596444661 podStartE2EDuration="7.921460508s" totalImagesPullingTime="5.325015847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:21.921175033 +0000 UTC m=+65.793733254" watchObservedRunningTime="2026-07-14 18:55:21.921460508 +0000 UTC m=+65.794018729"
Jul 14 18:55:22.283419 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.283354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:22.286739 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.286705    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 18:55:22.296747 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.296715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/139756f3-c948-4ecc-b8e1-a222e73af648-original-pull-secret\") pod \"global-pull-secret-syncer-ch6sn\" (UID: \"139756f3-c948-4ecc-b8e1-a222e73af648\") " pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:22.337053 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.336743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ch6sn"
Jul 14 18:55:22.384937 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.384303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:22.384937 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.384361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:22.387544 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.387517    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 18:55:22.387544 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.387518    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 18:55:22.397586 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.397556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d552c277-89f4-4fec-8e29-1cd663363215-metrics-certs\") pod \"network-metrics-daemon-jnqtz\" (UID: \"d552c277-89f4-4fec-8e29-1cd663363215\") " pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:22.397796 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.397776    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 18:55:22.408620 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.408589    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2wl\" (UniqueName: \"kubernetes.io/projected/8b167de1-abb3-4ce0-944b-b1731d194365-kube-api-access-7c2wl\") pod \"network-check-target-qg9bg\" (UID: \"8b167de1-abb3-4ce0-944b-b1731d194365\") " pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:22.428797 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.428771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zn8zd\""
Jul 14 18:55:22.433233 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.433212    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ltzph\""
Jul 14 18:55:22.436600 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.436581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:22.440899 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.440882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jnqtz"
Jul 14 18:55:22.906236 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.906192    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:22.911990 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:22.911961    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gqzm8"
Jul 14 18:55:23.154106 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.154051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:23.181003 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.180969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qg9bg"]
Jul 14 18:55:23.185585 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:23.185394    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b167de1_abb3_4ce0_944b_b1731d194365.slice/crio-84fc5587c81e187436f727f37a850cf1ef16f02337ac55c9da330257138d6151 WatchSource:0}: Error finding container 84fc5587c81e187436f727f37a850cf1ef16f02337ac55c9da330257138d6151: Status 404 returned error can't find the container with id 84fc5587c81e187436f727f37a850cf1ef16f02337ac55c9da330257138d6151
Jul 14 18:55:23.196173 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.196154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jnqtz"]
Jul 14 18:55:23.198478 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:23.198448    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd552c277_89f4_4fec_8e29_1cd663363215.slice/crio-6cc2639d3cb3572d59c63413cfa0a1f4c1c81ad9d97855bd75703955862c6901 WatchSource:0}: Error finding container 6cc2639d3cb3572d59c63413cfa0a1f4c1c81ad9d97855bd75703955862c6901: Status 404 returned error can't find the container with id 6cc2639d3cb3572d59c63413cfa0a1f4c1c81ad9d97855bd75703955862c6901
Jul 14 18:55:23.212225 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.212202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ch6sn"]
Jul 14 18:55:23.214624 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:23.214599    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139756f3_c948_4ecc_b8e1_a222e73af648.slice/crio-a71b602363a0377a74631aea9733ff3b2b0f50119857b5209fe47c89c02808ca WatchSource:0}: Error finding container a71b602363a0377a74631aea9733ff3b2b0f50119857b5209fe47c89c02808ca: Status 404 returned error can't find the container with id a71b602363a0377a74631aea9733ff3b2b0f50119857b5209fe47c89c02808ca
Jul 14 18:55:23.223847 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.223826    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:23.528178 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.528149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:23.638484 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.638448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:23.913938 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.913851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-96rqs" event={"ID":"f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9","Type":"ContainerStarted","Data":"12757082e8e2acf21a5bb7f6d70b029f2289aab9f76259fb8359b0c50373d60b"}
Jul 14 18:55:23.915314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.915281    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ch6sn" event={"ID":"139756f3-c948-4ecc-b8e1-a222e73af648","Type":"ContainerStarted","Data":"a71b602363a0377a74631aea9733ff3b2b0f50119857b5209fe47c89c02808ca"}
Jul 14 18:55:23.916909 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.916872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qg9bg" event={"ID":"8b167de1-abb3-4ce0-944b-b1731d194365","Type":"ContainerStarted","Data":"84fc5587c81e187436f727f37a850cf1ef16f02337ac55c9da330257138d6151"}
Jul 14 18:55:23.918659 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.918633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnqtz" event={"ID":"d552c277-89f4-4fec-8e29-1cd663363215","Type":"ContainerStarted","Data":"6cc2639d3cb3572d59c63413cfa0a1f4c1c81ad9d97855bd75703955862c6901"}
Jul 14 18:55:23.933577 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.933430    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-96rqs" podStartSLOduration=2.538423882 podStartE2EDuration="11.93339257s" totalImagesPullingTime="9.394968688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:55:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:23.932107505 +0000 UTC m=+67.804665733" watchObservedRunningTime="2026-07-14 18:55:23.93339257 +0000 UTC m=+67.805950787"
Jul 14 18:55:23.997646 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:23.997604    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:24.097429 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:24.097377    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:24.925486 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:24.925452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnqtz" event={"ID":"d552c277-89f4-4fec-8e29-1cd663363215","Type":"ContainerStarted","Data":"d30e070197644760cd5cc8c8a35c077b4de63f7380ddac9fe50c2c53d0a19e97"}
Jul 14 18:55:24.925904 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:24.925509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jnqtz" event={"ID":"d552c277-89f4-4fec-8e29-1cd663363215","Type":"ContainerStarted","Data":"388660485ef35593b478361f8af6ea99977b444128e7e404afb6da137a1149ca"}
Jul 14 18:55:24.948976 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:24.948925    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jnqtz" podStartSLOduration=68.071856672 podStartE2EDuration="1m8.948909675s" totalImagesPullingTime="877.053003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:24.948671743 +0000 UTC m=+68.821229964" watchObservedRunningTime="2026-07-14 18:55:24.948909675 +0000 UTC m=+68.821467895"
Jul 14 18:55:25.076730 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:25.076690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:25.177125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:25.177048    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:25.268207 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:25.268157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:27.905989 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:27.905955    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-96h9g"
Jul 14 18:55:28.815853 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.815810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:28.917381 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.917351    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:28.940497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.940462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ch6sn" event={"ID":"139756f3-c948-4ecc-b8e1-a222e73af648","Type":"ContainerStarted","Data":"1ab82e608025e9a5ac5abf9a90e2b89ed9b327fc762b0e7389555f4d594c205d"}
Jul 14 18:55:28.941896 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.941867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qg9bg" event={"ID":"8b167de1-abb3-4ce0-944b-b1731d194365","Type":"ContainerStarted","Data":"cc079b6a375eaf28a42ad1abeb12492dec779d59149cbac7803cadcc3dd1eeec"}
Jul 14 18:55:28.942023 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.942010    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:55:28.958499 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.958447    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ch6sn" podStartSLOduration=68.018161238 podStartE2EDuration="1m12.958429479s" totalImagesPullingTime="4.940268241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:28.956993364 +0000 UTC m=+72.829551584" watchObservedRunningTime="2026-07-14 18:55:28.958429479 +0000 UTC m=+72.830987690"
Jul 14 18:55:28.976035 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:28.975982    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qg9bg" podStartSLOduration=68.009068594 podStartE2EDuration="1m12.975968108s" totalImagesPullingTime="4.966899514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:28.975426402 +0000 UTC m=+72.847984613" watchObservedRunningTime="2026-07-14 18:55:28.975968108 +0000 UTC m=+72.848526328"
Jul 14 18:55:29.845422 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:29.845359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:30.760333 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:30.760294    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:31.075317 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.075235    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:31.650212 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.650180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:31.742126 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.742088    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:31.842168 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.842134    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:31.925423 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.925315    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"]
Jul 14 18:55:31.927647 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.927624    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:31.930383 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.930358    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 18:55:31.930515 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.930453    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 18:55:31.931490 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.931472    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 18:55:31.931603 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.931570    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 18:55:31.931780 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.931762    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 18:55:31.931906 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.931851    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k8pwp\""
Jul 14 18:55:31.932043 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.932024    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:31.935931 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.935914    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nrpjf"]
Jul 14 18:55:31.938122 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.938098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:31.938671 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.938654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"]
Jul 14 18:55:31.943772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.943751    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 18:55:31.943868 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.943793    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 18:55:31.944057 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.944036    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rkcqk\""
Jul 14 18:55:31.944130 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.944093    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 18:55:31.948255 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.948236    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"]
Jul 14 18:55:31.950745 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.950715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:31.953737 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.953721    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 18:55:31.953737 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.953731    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 18:55:31.954207 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.954191    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fjvqq\""
Jul 14 18:55:31.954690 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.954672    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 18:55:31.965160 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:31.965138    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"]
Jul 14 18:55:32.024544 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.024509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.055533 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.055637 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055549    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/603ef2e5-892c-428c-87e0-c797daa806c9-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.055637 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-sys\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.055637 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-textfile\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.055817 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603ef2e5-892c-428c-87e0-c797daa806c9-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.055817 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.055817 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a320ccfe-ab58-473e-ad7f-b38904672c73-metrics-client-ca\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.055817 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-tls\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.055995 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.055995 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnc2k\" (UniqueName: \"kubernetes.io/projected/603ef2e5-892c-428c-87e0-c797daa806c9-kube-api-access-bnc2k\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.055995 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.055995 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.055951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvbs\" (UniqueName: \"kubernetes.io/projected/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-kube-api-access-vfvbs\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.056141 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dtkh\" (UniqueName: \"kubernetes.io/projected/a320ccfe-ab58-473e-ad7f-b38904672c73-kube-api-access-9dtkh\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.056141 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.056141 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.056141 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.056141 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-wtmp\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.056314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.056314 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.056186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-root\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.105071 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.105038    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.157485 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.157485 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnc2k\" (UniqueName: \"kubernetes.io/projected/603ef2e5-892c-428c-87e0-c797daa806c9-kube-api-access-bnc2k\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvbs\" (UniqueName: \"kubernetes.io/projected/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-kube-api-access-vfvbs\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dtkh\" (UniqueName: \"kubernetes.io/projected/a320ccfe-ab58-473e-ad7f-b38904672c73-kube-api-access-9dtkh\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.157661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.157721 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:32.157676    2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 14 18:55:32.158065 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:32.157868    2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 14 18:55:32.158065 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:32.158001    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-tls podName:f1ccaaeb-5935-43d3-b92e-765e26cde0a1 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:32.657973304 +0000 UTC m=+76.530531503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-5lvz9" (UID: "f1ccaaeb-5935-43d3-b92e-765e26cde0a1") : secret "openshift-state-metrics-tls" not found
Jul 14 18:55:32.158065 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:55:32.158032    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-tls podName:603ef2e5-892c-428c-87e0-c797daa806c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:32.658022123 +0000 UTC m=+76.530580330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-zdfnf" (UID: "603ef2e5-892c-428c-87e0-c797daa806c9") : secret "kube-state-metrics-tls" not found
Jul 14 18:55:32.158191 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-wtmp\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158191 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158191 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-root\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158320 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158320 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/603ef2e5-892c-428c-87e0-c797daa806c9-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.158320 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-sys\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-sys\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-root\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-textfile\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603ef2e5-892c-428c-87e0-c797daa806c9-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.158497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.158497 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a320ccfe-ab58-473e-ad7f-b38904672c73-metrics-client-ca\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158776 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-tls\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158776 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-textfile\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.158776 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.159088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/603ef2e5-892c-428c-87e0-c797daa806c9-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.159088 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.158941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-wtmp\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.159301 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.159258    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.159593 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.159548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/603ef2e5-892c-428c-87e0-c797daa806c9-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.159721 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.159700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a320ccfe-ab58-473e-ad7f-b38904672c73-metrics-client-ca\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.160014 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.159996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.160744 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.160716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.160845 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.160825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.161106 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.161088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.161385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.161367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a320ccfe-ab58-473e-ad7f-b38904672c73-node-exporter-tls\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.166839 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.166813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dtkh\" (UniqueName: \"kubernetes.io/projected/a320ccfe-ab58-473e-ad7f-b38904672c73-kube-api-access-9dtkh\") pod \"node-exporter-nrpjf\" (UID: \"a320ccfe-ab58-473e-ad7f-b38904672c73\") " pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.167114 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.167075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnc2k\" (UniqueName: \"kubernetes.io/projected/603ef2e5-892c-428c-87e0-c797daa806c9-kube-api-access-bnc2k\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.167292 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.167274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvbs\" (UniqueName: \"kubernetes.io/projected/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-kube-api-access-vfvbs\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.170671 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.170641    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.246948 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.246915    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrpjf"
Jul 14 18:55:32.256642 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:32.256616    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda320ccfe_ab58_473e_ad7f_b38904672c73.slice/crio-cb1bde473a3cde1ca6c82c2b3e0178b2a37aa1fad746f6e46cb8503f820fffb6 WatchSource:0}: Error finding container cb1bde473a3cde1ca6c82c2b3e0178b2a37aa1fad746f6e46cb8503f820fffb6: Status 404 returned error can't find the container with id cb1bde473a3cde1ca6c82c2b3e0178b2a37aa1fad746f6e46cb8503f820fffb6
Jul 14 18:55:32.269207 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.269182    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.370044 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.370010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.470562 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.470523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.533280 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.533191    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.662687 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.662651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.662848 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.662694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.665225 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.665194    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1ccaaeb-5935-43d3-b92e-765e26cde0a1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-5lvz9\" (UID: \"f1ccaaeb-5935-43d3-b92e-765e26cde0a1\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.665225 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.665193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/603ef2e5-892c-428c-87e0-c797daa806c9-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zdfnf\" (UID: \"603ef2e5-892c-428c-87e0-c797daa806c9\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.836443 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.836337    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"
Jul 14 18:55:32.859096 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.859067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"
Jul 14 18:55:32.899123 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.899069    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:32.962925 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.962883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrpjf" event={"ID":"a320ccfe-ab58-473e-ad7f-b38904672c73","Type":"ContainerStarted","Data":"cb1bde473a3cde1ca6c82c2b3e0178b2a37aa1fad746f6e46cb8503f820fffb6"}
Jul 14 18:55:32.988708 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.988686    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9"]
Jul 14 18:55:32.991135 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:32.991104    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ccaaeb_5935_43d3_b92e_765e26cde0a1.slice/crio-841617adcd49c50e3fa75a217e826291737475f91fd0107f35cf63980bc9ec18 WatchSource:0}: Error finding container 841617adcd49c50e3fa75a217e826291737475f91fd0107f35cf63980bc9ec18: Status 404 returned error can't find the container with id 841617adcd49c50e3fa75a217e826291737475f91fd0107f35cf63980bc9ec18
Jul 14 18:55:32.992538 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:32.992510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:33.007756 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:33.007704    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zdfnf"]
Jul 14 18:55:33.009964 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:33.009932    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603ef2e5_892c_428c_87e0_c797daa806c9.slice/crio-3022685493b81901cb131a22b55927bc5e8ed845d8a5bea37a93bc50ca3656a3 WatchSource:0}: Error finding container 3022685493b81901cb131a22b55927bc5e8ed845d8a5bea37a93bc50ca3656a3: Status 404 returned error can't find the container with id 3022685493b81901cb131a22b55927bc5e8ed845d8a5bea37a93bc50ca3656a3
Jul 14 18:55:33.232776 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:33.232743    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:33.969311 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:33.969216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf" event={"ID":"603ef2e5-892c-428c-87e0-c797daa806c9","Type":"ContainerStarted","Data":"3022685493b81901cb131a22b55927bc5e8ed845d8a5bea37a93bc50ca3656a3"}
Jul 14 18:55:33.972925 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:33.972799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9" event={"ID":"f1ccaaeb-5935-43d3-b92e-765e26cde0a1","Type":"ContainerStarted","Data":"050763ddfdc99020eebb72e9a4b9f9d0142e7bc5bdef1f57b857a39323d5ac0c"}
Jul 14 18:55:33.972925 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:33.972861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9" event={"ID":"f1ccaaeb-5935-43d3-b92e-765e26cde0a1","Type":"ContainerStarted","Data":"e656a1417eb1e1133b34a6b262d66976b9c869b1d819998e4a602053035e6115"}
Jul 14 18:55:33.972925 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:33.972888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9" event={"ID":"f1ccaaeb-5935-43d3-b92e-765e26cde0a1","Type":"ContainerStarted","Data":"841617adcd49c50e3fa75a217e826291737475f91fd0107f35cf63980bc9ec18"}
Jul 14 18:55:34.072898 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.072807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:34.725685 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.725655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:34.829163 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.829128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:34.876044 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.875997    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78b49755bc-gtvpd"
Jul 14 18:55:34.909209 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.909181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:34.944440 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.940680    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"]
Jul 14 18:55:34.946889 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.946862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:34.950954 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.950645    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 14 18:55:34.950954 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.950725    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 14 18:55:34.951230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.951214    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-771odhv5i38i6\""
Jul 14 18:55:34.952166 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.951736    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 14 18:55:34.955870 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.955296    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 14 18:55:34.955870 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.955579    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 14 18:55:34.958220 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.958181    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-klz69\""
Jul 14 18:55:34.973879 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.973835    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"]
Jul 14 18:55:34.980786 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.980708    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf" event={"ID":"603ef2e5-892c-428c-87e0-c797daa806c9","Type":"ContainerStarted","Data":"0026a94ad28bdbe37ebe050e3527a0f1c2eff1b0ae8acba6e12bbb468b39eb70"}
Jul 14 18:55:34.980786 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.980765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf" event={"ID":"603ef2e5-892c-428c-87e0-c797daa806c9","Type":"ContainerStarted","Data":"11c109048d47e9b2c108f7381a82d7936fa4c4184979778aaac02217d5b76e43"}
Jul 14 18:55:34.984009 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.983984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9" event={"ID":"f1ccaaeb-5935-43d3-b92e-765e26cde0a1","Type":"ContainerStarted","Data":"9420ad73be28a33e4bdff5abe3e08553f90a0cf8952d0b04c1fd562a0ed73dd9"}
Jul 14 18:55:34.985635 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.985608    2562 generic.go:356] "Generic (PLEG): container finished" podID="a320ccfe-ab58-473e-ad7f-b38904672c73" containerID="623427e8461d547faff7935edb9bfddf2f66dc6f08d9e57da1d7b742ee146f03" exitCode=0
Jul 14 18:55:34.985751 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:34.985700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrpjf" event={"ID":"a320ccfe-ab58-473e-ad7f-b38904672c73","Type":"ContainerDied","Data":"623427e8461d547faff7935edb9bfddf2f66dc6f08d9e57da1d7b742ee146f03"}
Jul 14 18:55:35.079696 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.078714    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-5lvz9" podStartSLOduration=2.422905091 podStartE2EDuration="4.078690268s" totalImagesPullingTime="1.655785177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:35.07661987 +0000 UTC m=+78.949178092" watchObservedRunningTime="2026-07-14 18:55:35.078690268 +0000 UTC m=+78.951248489"
Jul 14 18:55:35.082980 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.082954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.083146 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.083122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.084554 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.083637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.084554 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.083715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.084554 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.083760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-tls\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.084554 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.083853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-metrics-client-ca\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.084809 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.084352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-grpc-tls\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.084809 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.084749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrwh\" (UniqueName: \"kubernetes.io/projected/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-kube-api-access-dnrwh\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.132283 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.132247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:35.186694 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.186888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-tls\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.186888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-metrics-client-ca\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.186888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-grpc-tls\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.186888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrwh\" (UniqueName: \"kubernetes.io/projected/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-kube-api-access-dnrwh\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.186888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.186888 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.187185 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.186908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.187825 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.187597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-metrics-client-ca\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.189940 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.189910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.190308 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.190284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.190616 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.190466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-tls\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.190616 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.190466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.190723 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.190612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-grpc-tls\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.191477 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.191457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.197713 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.197693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrwh\" (UniqueName: \"kubernetes.io/projected/2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4-kube-api-access-dnrwh\") pod \"thanos-querier-6c6f56d6c-mnc26\" (UID: \"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4\") " pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.226562 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.226534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:35.291702 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.291618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:35.321541 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.321500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:35.397972 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.397941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:35.471190 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.471164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"]
Jul 14 18:55:35.472910 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:35.472882    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc64e0a_bde8_4094_a2b6_b0dd3ee74cd4.slice/crio-9e000830d94e9b05994ad73cd88d6e836100f723b907f75d37689b0b3eb199ee WatchSource:0}: Error finding container 9e000830d94e9b05994ad73cd88d6e836100f723b907f75d37689b0b3eb199ee: Status 404 returned error can't find the container with id 9e000830d94e9b05994ad73cd88d6e836100f723b907f75d37689b0b3eb199ee
Jul 14 18:55:35.991557 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.991513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"9e000830d94e9b05994ad73cd88d6e836100f723b907f75d37689b0b3eb199ee"}
Jul 14 18:55:35.993681 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.993654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf" event={"ID":"603ef2e5-892c-428c-87e0-c797daa806c9","Type":"ContainerStarted","Data":"83b54fbfb94c8236499d7dd188e4d98556a1708322cf0795d08bf88596ec0df0"}
Jul 14 18:55:35.995578 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.995552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrpjf" event={"ID":"a320ccfe-ab58-473e-ad7f-b38904672c73","Type":"ContainerStarted","Data":"beae987ac76c838cbb1d28c2fbf3f500ace296caeeb4073132eb64462f1bbf26"}
Jul 14 18:55:35.995688 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:35.995598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrpjf" event={"ID":"a320ccfe-ab58-473e-ad7f-b38904672c73","Type":"ContainerStarted","Data":"1a4d7b8345d2d726813bb1b3ec70d10cc71478695bdcd9c6c88694f7cbcfa089"}
Jul 14 18:55:36.021853 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.021808    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-zdfnf" podStartSLOduration=3.2754061930000002 podStartE2EDuration="5.021792837s" totalImagesPullingTime="1.746386644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:36.021526157 +0000 UTC m=+79.894084383" watchObservedRunningTime="2026-07-14 18:55:36.021792837 +0000 UTC m=+79.894351087"
Jul 14 18:55:36.056453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.056334    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nrpjf" podStartSLOduration=2.453442281 podStartE2EDuration="5.056316388s" totalImagesPullingTime="2.501999354s" totalInitContainerRuntime="100.874753ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:36.05403923 +0000 UTC m=+79.926597452" watchObservedRunningTime="2026-07-14 18:55:36.056316388 +0000 UTC m=+79.928874607"
Jul 14 18:55:36.173659 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.173619    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:36.258238 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.258204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:36.382205 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.382114    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:36.482480 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.482445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:36.725568 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.725538    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:36.792988 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:36.792952    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:37.491114 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:37.491002    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:37.810427 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:37.810321    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:37.894668 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:37.894606    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:38.093123 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.093031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:38.203649 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.203614    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:38.239156 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.239116    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:55:38.241991 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.241968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.245195 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.245163    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 18:55:38.245537 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.245510    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 18:55:38.245537 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.245515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 18:55:38.245769 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.245515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4khd\""
Jul 14 18:55:38.245769 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.245515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 18:55:38.245769 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.245669    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 18:55:38.246020 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.246004    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 18:55:38.246104 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.246024    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 18:55:38.246104 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.246036    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 18:55:38.246104 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.246062    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 18:55:38.246104 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.246037    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 18:55:38.246372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.246352    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 18:55:38.247276 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.247256    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 18:55:38.247382 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.247261    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 18:55:38.247847 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.247828    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dqk2ge428oaln\""
Jul 14 18:55:38.250769 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.250686    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 18:55:38.252241 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.252194    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 18:55:38.261174 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.261150    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:55:38.304030 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.303993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:38.310592 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-config\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310592 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310592 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310592 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310592 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-config-out\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310796    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.310923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310912    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-web-config\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.310986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.311013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.311046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.311069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.311089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbgrm\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-kube-api-access-jbgrm\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.311157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.311372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.311193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.411965 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.411868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.411965 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.411917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.411965 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.411935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.411965 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.411950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.411974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.411997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-config-out\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412130    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-web-config\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbgrm\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-kube-api-access-jbgrm\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.412295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413073 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413073 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-config\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413073 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413073 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413073 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413073 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.412885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.413382 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.413307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.415000 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.413590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.415000 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.414109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.415000 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.414881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.415223 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.415067    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-config-out\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.415571 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.415535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.416346 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.416320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.416543 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.416520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.417132 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.417110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.417237 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.417154    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-config\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.417765 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.417741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.417939 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.417921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.418076 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.418058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.418337 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.418317    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.419222 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.419203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-web-config\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.419298 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.419281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.423237 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.423218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbgrm\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-kube-api-access-jbgrm\") pod \"prometheus-k8s-0\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.553722 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.553680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:38.702816 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.702727    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:55:38.962100 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:38.962010    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:39.067085 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:39.067046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:39.143942 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:55:39.143905    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a6350b_7bb0_4296_9319_3e96936407bb.slice/crio-81632b297523360ac3600155bf912aff17fd360b124b4d46724acd5243675287 WatchSource:0}: Error finding container 81632b297523360ac3600155bf912aff17fd360b124b4d46724acd5243675287: Status 404 returned error can't find the container with id 81632b297523360ac3600155bf912aff17fd360b124b4d46724acd5243675287
Jul 14 18:55:39.163257 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:39.163217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:40.017794 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:40.017547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"6741e7c78b9743144e1885daf98324becc1d37f059c0e43be173aefef1d9aab7"}
Jul 14 18:55:40.017794 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:40.017679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"1c9470c6de4e8210116820e3249584442daa4c6a83fcac57689b02ae7e14fd94"}
Jul 14 18:55:40.017794 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:40.017728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"7ef45ad2d0d63d6484cc03922f8b8aa70f50dad70b6aa18dd9fd65c04e916be6"}
Jul 14 18:55:40.018955 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:40.018929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"81632b297523360ac3600155bf912aff17fd360b124b4d46724acd5243675287"}
Jul 14 18:55:41.026136 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.026104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"583cfab0772415e78adae9a797db1d9715d32a48a5a5a371aaba07d4e5696017"}
Jul 14 18:55:41.026615 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.026185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"0e7b4aede91dc4bec7b31dd39c55e37150e4a889d9239364a732e0141b3b2398"}
Jul 14 18:55:41.026615 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.026230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" event={"ID":"2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4","Type":"ContainerStarted","Data":"40219acbfd1ea13f3009465d7445eb21b78b5973f115e326803ac7dd161e51bd"}
Jul 14 18:55:41.026615 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.026274    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:41.027735 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.027709    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be" exitCode=0
Jul 14 18:55:41.027868 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.027747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"}
Jul 14 18:55:41.052767 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:41.052709    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26" podStartSLOduration=2.428610962 podStartE2EDuration="7.052688485s" totalImagesPullingTime="4.624077523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:41.05123915 +0000 UTC m=+84.923797411" watchObservedRunningTime="2026-07-14 18:55:41.052688485 +0000 UTC m=+84.925246706"
Jul 14 18:55:42.222915 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:42.222882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:42.322168 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:42.322136    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:45.049809 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:45.049773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"}
Jul 14 18:55:45.050179 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:45.049878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"}
Jul 14 18:55:45.050179 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:45.049921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"}
Jul 14 18:55:45.050179 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:45.049959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"}
Jul 14 18:55:45.191256 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:45.191220    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:45.261491 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:45.261389    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:46.057738 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:46.057706    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"}
Jul 14 18:55:46.058140 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:46.057776    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerStarted","Data":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"}
Jul 14 18:55:46.094786 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:46.094721    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.253222347 podStartE2EDuration="8.094701312s" totalImagesPullingTime="4.820523641s" totalInitContainerRuntime="20.955324ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:46.093040711 +0000 UTC m=+89.965598934" watchObservedRunningTime="2026-07-14 18:55:46.094701312 +0000 UTC m=+89.967259532"
Jul 14 18:55:46.877476 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:46.877443    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:47.037882 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:47.037845    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c6f56d6c-mnc26"
Jul 14 18:55:48.197385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:48.197350    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:48.554084 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:48.554041    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:55:49.238148 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:49.238113    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:49.301709 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:49.301675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:49.744241 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:49.744208    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:49.811687 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:49.811652    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:53.237784 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:53.237737    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:53.337747 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:53.337718    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:56.334676 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:56.334635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:56.438540 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:56.438499    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:56.502665 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:56.502635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:55:59.947776 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:55:59.947746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qg9bg"
Jul 14 18:56:04.730608 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:04.730571    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:04.831755 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:04.831722    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:31.569423 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:31.569354    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:31.671373 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:31.671339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:38.554734 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:38.554689    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:38.575049 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:38.575019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:39.268801 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:39.268776    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:48.084801 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:48.084765    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:48.147202 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:48.147164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:51.678633 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:51.678599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:51.745352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:51.745318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:51.812018 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:51.811983    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:51.876116 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:51.876083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:52.438014 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:52.437982    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:52.561877 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:52.561849    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:56:56.648348 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648314    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:56:56.648856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648612    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="prometheus" containerID="cri-o://79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3" gracePeriod=600
Jul 14 18:56:56.648856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648654    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="config-reloader" containerID="cri-o://4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88" gracePeriod=600
Jul 14 18:56:56.648856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648659    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f" gracePeriod=600
Jul 14 18:56:56.648856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648683    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="thanos-sidecar" containerID="cri-o://0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea" gracePeriod=600
Jul 14 18:56:56.648856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648667    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-web" containerID="cri-o://abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b" gracePeriod=600
Jul 14 18:56:56.648856 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.648623    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy" containerID="cri-o://a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523" gracePeriod=600
Jul 14 18:56:56.906440 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:56.906365    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.066451 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066386    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-2\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066451 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066459    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-config-out\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066481    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-config\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-thanos-prometheus-http-client-file\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-1\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066540    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-tls-assets\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066573    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-serving-certs-ca-bundle\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066621    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-grpc-tls\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066650    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.066683 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-kubelet-serving-ca-bundle\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066707    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-metrics-client-certs\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066736    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-trusted-ca-bundle\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066759    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbgrm\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-kube-api-access-jbgrm\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066793    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-web-config\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066820    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-0\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066843    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-tls\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-kube-rbac-proxy\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066924    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066960    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-db\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066963    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.066988    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-metrics-client-ca\") pod \"03a6350b-7bb0-4296-9319-3e96936407bb\" (UID: \"03a6350b-7bb0-4296-9319-3e96936407bb\") "
Jul 14 18:56:57.067074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.067059    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.067872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.067316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.067872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.067646    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-metrics-client-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.067872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.067669    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.067872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.067687    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.067872 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.067397    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.069131 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.069103    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:56:57.070206 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.069574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.070206 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.070093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.070206 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.069693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-config-out" (OuterVolumeSpecName: "config-out") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:56:57.070206 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.070162    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.070206 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.070178    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-config" (OuterVolumeSpecName: "config") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.070566 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.070548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:56:57.070646 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.070565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.070968 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.070938    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.071242 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.071207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:56:57.071776 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.071756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.072107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.072080    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.072385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.072368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.072486 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.072381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-kube-api-access-jbgrm" (OuterVolumeSpecName: "kube-api-access-jbgrm") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "kube-api-access-jbgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:56:57.072486 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.072392    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.082081 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.082058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-web-config" (OuterVolumeSpecName: "web-config") pod "03a6350b-7bb0-4296-9319-3e96936407bb" (UID: "03a6350b-7bb0-4296-9319-3e96936407bb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:56:57.168097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168016    2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-grpc-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168046    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168057    2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168069    2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-metrics-client-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168078    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168097 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168086    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbgrm\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-kube-api-access-jbgrm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168115    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-web-config\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168125    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168134    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168142    2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-kube-rbac-proxy\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168151    2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168159    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-db\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168166    2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/03a6350b-7bb0-4296-9319-3e96936407bb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168174    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03a6350b-7bb0-4296-9319-3e96936407bb-config-out\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168181    2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-config\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168189    2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/03a6350b-7bb0-4296-9319-3e96936407bb-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.168368 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.168196    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03a6350b-7bb0-4296-9319-3e96936407bb-tls-assets\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 18:56:57.328125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328093    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f" exitCode=0
Jul 14 18:56:57.328125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328118    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523" exitCode=0
Jul 14 18:56:57.328125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328124    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b" exitCode=0
Jul 14 18:56:57.328125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328131    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea" exitCode=0
Jul 14 18:56:57.328125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328136    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88" exitCode=0
Jul 14 18:56:57.328125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328141    2562 generic.go:356] "Generic (PLEG): container finished" podID="03a6350b-7bb0-4296-9319-3e96936407bb" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3" exitCode=0
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"}
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328185    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"}
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"}
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"}
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328337    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"}
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328368    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"}
Jul 14 18:56:57.328453 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.328391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"03a6350b-7bb0-4296-9319-3e96936407bb","Type":"ContainerDied","Data":"81632b297523360ac3600155bf912aff17fd360b124b4d46724acd5243675287"}
Jul 14 18:56:57.336853 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.336835    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.343841 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.343821    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.350384 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.350363    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.354694 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.354673    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:56:57.358252 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.358237    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.358962 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.358943    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:56:57.364968 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.364952    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.372129 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.372111    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.379125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.379107    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.379429 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.379390    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.379487 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.379442    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"} err="failed to get container status \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist"
Jul 14 18:56:57.379526 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.379487    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.379751 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.379731    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.379809 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.379762    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"} err="failed to get container status \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist"
Jul 14 18:56:57.379809 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.379785    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.380033 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.380017    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.380070 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380039    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"} err="failed to get container status \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist"
Jul 14 18:56:57.380070 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380054    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.380275 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.380260    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.380321 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380279    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"} err="failed to get container status \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist"
Jul 14 18:56:57.380321 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380292    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.380511 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.380495    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.380556 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380515    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"} err="failed to get container status \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist"
Jul 14 18:56:57.380556 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380528    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.380757 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.380742    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.380800 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380762    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"} err="failed to get container status \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist"
Jul 14 18:56:57.380800 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380776    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.380964 ip-10-0-132-123 kubenswrapper[2562]: E0714 18:56:57.380948    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.381003 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380967    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"} err="failed to get container status \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist"
Jul 14 18:56:57.381003 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.380979    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.381170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381146    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"} err="failed to get container status \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist"
Jul 14 18:56:57.381170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381163    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.381376 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381358    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"} err="failed to get container status \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist"
Jul 14 18:56:57.381439 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381378    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.381623 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381606    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"} err="failed to get container status \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist"
Jul 14 18:56:57.381694 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381626    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.381836 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381822    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"} err="failed to get container status \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist"
Jul 14 18:56:57.381881 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.381837    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.382040 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382023    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"} err="failed to get container status \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist"
Jul 14 18:56:57.382105 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382041    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.382236 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382217    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"} err="failed to get container status \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist"
Jul 14 18:56:57.382278 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382237    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.382451 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382433    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"} err="failed to get container status \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist"
Jul 14 18:56:57.382522 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382453    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.382701 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382684    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"} err="failed to get container status \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist"
Jul 14 18:56:57.382758 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382701    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.382905 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382889    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"} err="failed to get container status \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist"
Jul 14 18:56:57.382905 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.382907    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.383080 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383062    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"} err="failed to get container status \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist"
Jul 14 18:56:57.383140 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383082    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.383284 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383266    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"} err="failed to get container status \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist"
Jul 14 18:56:57.383319 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383285    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.383493 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383477    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"} err="failed to get container status \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist"
Jul 14 18:56:57.383540 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383494    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.383712 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383694    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"} err="failed to get container status \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist"
Jul 14 18:56:57.383766 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383713    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.383877 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383860    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"} err="failed to get container status \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist"
Jul 14 18:56:57.383919 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.383877    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.384051 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384030    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"} err="failed to get container status \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist"
Jul 14 18:56:57.384119 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384053    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.384239 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384223    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"} err="failed to get container status \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist"
Jul 14 18:56:57.384280 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384240    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.384523 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384503    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"} err="failed to get container status \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist"
Jul 14 18:56:57.384582 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384572    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.384804 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384780    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"} err="failed to get container status \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist"
Jul 14 18:56:57.384804 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.384801    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.385221 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.385192    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"} err="failed to get container status \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist"
Jul 14 18:56:57.385337 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.385221    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.385576 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.385549    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"} err="failed to get container status \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist"
Jul 14 18:56:57.385576 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.385576    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.385865 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.385829    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"} err="failed to get container status \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist"
Jul 14 18:56:57.385865 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.385865    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.386184 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386158    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"} err="failed to get container status \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist"
Jul 14 18:56:57.386258 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386185    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.386518 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386391    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"} err="failed to get container status \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist"
Jul 14 18:56:57.386518 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386520    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.386766 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386750    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"} err="failed to get container status \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist"
Jul 14 18:56:57.386826 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386766    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.386986 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386965    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"} err="failed to get container status \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist"
Jul 14 18:56:57.387072 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.386988    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.387235 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387220    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:56:57.387319 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387213    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"} err="failed to get container status \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist"
Jul 14 18:56:57.387359 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387321    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.387556 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387542    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy"
Jul 14 18:56:57.387610 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387552    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"} err="failed to get container status \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist"
Jul 14 18:56:57.387610 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387570    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.387610 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387560    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387615    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="prometheus"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387622    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="prometheus"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387633    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="thanos-sidecar"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387639    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="thanos-sidecar"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387652    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-web"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387661    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-web"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387669    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="config-reloader"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387674    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="config-reloader"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387680    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="init-config-reloader"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387685    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="init-config-reloader"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387695    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-thanos"
Jul 14 18:56:57.387724 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387703    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-thanos"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387767    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="prometheus"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387775    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="thanos-sidecar"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387781    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-web"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387787    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="config-reloader"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387793    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387799    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" containerName="kube-rbac-proxy-thanos"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387808    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"} err="failed to get container status \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.387833    2562 scope.go:117] "RemoveContainer" containerID="7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388050    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f"} err="failed to get container status \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": rpc error: code = NotFound desc = could not find container \"7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f\": container with ID starting with 7246d2af35e2546208696bc18e416d82d631026908da46b2c453a3c8bf71432f not found: ID does not exist"
Jul 14 18:56:57.388150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388064    2562 scope.go:117] "RemoveContainer" containerID="a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"
Jul 14 18:56:57.388522 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388276    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523"} err="failed to get container status \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": rpc error: code = NotFound desc = could not find container \"a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523\": container with ID starting with a3bb28ec5d5bfa35f294d73ed02d6b8a49a8f72abd7aca1d83b5c535601eb523 not found: ID does not exist"
Jul 14 18:56:57.388522 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388307    2562 scope.go:117] "RemoveContainer" containerID="abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"
Jul 14 18:56:57.388595 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388532    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b"} err="failed to get container status \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": rpc error: code = NotFound desc = could not find container \"abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b\": container with ID starting with abe740ab6d33e6a7c0a73bbdc45737863af5640abe5ca065255cbe98f2e5a02b not found: ID does not exist"
Jul 14 18:56:57.388595 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388549    2562 scope.go:117] "RemoveContainer" containerID="0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"
Jul 14 18:56:57.388773 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388754    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea"} err="failed to get container status \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": rpc error: code = NotFound desc = could not find container \"0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea\": container with ID starting with 0d882717fd9dbd9c8ba98f65e0f4a763de6cd54b9428047c7b55f1c04ce2c6ea not found: ID does not exist"
Jul 14 18:56:57.388818 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388775    2562 scope.go:117] "RemoveContainer" containerID="4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"
Jul 14 18:56:57.389019 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.388997    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88"} err="failed to get container status \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": rpc error: code = NotFound desc = could not find container \"4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88\": container with ID starting with 4928ee7c4e5588bcecebf6d8b2bbb5d85e471d416b4c3ff07c412e30b8cfaa88 not found: ID does not exist"
Jul 14 18:56:57.389152 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.389021    2562 scope.go:117] "RemoveContainer" containerID="79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"
Jul 14 18:56:57.389282 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.389260    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3"} err="failed to get container status \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": rpc error: code = NotFound desc = could not find container \"79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3\": container with ID starting with 79d541e535e43190f3a348ccf722ca290f4e0faf41efbadc5446f0d4653b77e3 not found: ID does not exist"
Jul 14 18:56:57.389335 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.389285    2562 scope.go:117] "RemoveContainer" containerID="0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"
Jul 14 18:56:57.389548 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.389527    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be"} err="failed to get container status \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": rpc error: code = NotFound desc = could not find container \"0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be\": container with ID starting with 0fd0a225150a5bee48e6feabf3a8f8bf563cfca0f7876e98dfc7b34d087810be not found: ID does not exist"
Jul 14 18:56:57.393074 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.393060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.395822 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.395798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 18:56:57.395915 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.395798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4khd\""
Jul 14 18:56:57.395915 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.395894    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 18:56:57.396004 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.395924    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 18:56:57.396059 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396042    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 18:56:57.396115 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 18:56:57.396166 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 18:56:57.396166 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396159    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 18:56:57.396455 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396440    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 18:56:57.396455 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396439    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 18:56:57.396602 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396456    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 18:56:57.396602 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dqk2ge428oaln\""
Jul 14 18:56:57.396602 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396439    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 18:56:57.396759 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396719    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 18:56:57.397007 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.396987    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 18:56:57.398934 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.398914    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 18:56:57.402541 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.402521    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 18:56:57.412443 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.412394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:56:57.571715 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9e94bd1-5c75-4816-ae29-022f5b8152ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571715 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9e94bd1-5c75-4816-ae29-022f5b8152ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571768    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.571923 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lwv\" (UniqueName: \"kubernetes.io/projected/f9e94bd1-5c75-4816-ae29-022f5b8152ad-kube-api-access-86lwv\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.571964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-config\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572060    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572118    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.572170 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.572160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673017 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.672989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9e94bd1-5c75-4816-ae29-022f5b8152ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673385 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9e94bd1-5c75-4816-ae29-022f5b8152ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lwv\" (UniqueName: \"kubernetes.io/projected/f9e94bd1-5c75-4816-ae29-022f5b8152ad-kube-api-access-86lwv\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.673725 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674198 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-config\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674198 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674198 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.673921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674198 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.674010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674198 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.674075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674198 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.674150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.674535 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.674509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.676295 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.676268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f9e94bd1-5c75-4816-ae29-022f5b8152ad-config-out\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.676444 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.676426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-web-config\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.676772 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.676684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.677047 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.676881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.677234 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.677204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.677503 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.677324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-config\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.678085 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.677735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.678085 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.677756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.678085 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.677874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.678085 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.678017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.679016 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.678991    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f9e94bd1-5c75-4816-ae29-022f5b8152ad-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.679079 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.679068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.679520 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.679500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.679674 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.679660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9e94bd1-5c75-4816-ae29-022f5b8152ad-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.681671 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.681649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f9e94bd1-5c75-4816-ae29-022f5b8152ad-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.683325 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.683306    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lwv\" (UniqueName: \"kubernetes.io/projected/f9e94bd1-5c75-4816-ae29-022f5b8152ad-kube-api-access-86lwv\") pod \"prometheus-k8s-0\" (UID: \"f9e94bd1-5c75-4816-ae29-022f5b8152ad\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.703692 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.703661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:56:57.851816 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:57.851734    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 18:56:57.854874 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:56:57.854849    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e94bd1_5c75_4816_ae29_022f5b8152ad.slice/crio-b2f1dcc6808288c856424bb47b943f0a203cef578324483e6e628bb350134dcb WatchSource:0}: Error finding container b2f1dcc6808288c856424bb47b943f0a203cef578324483e6e628bb350134dcb: Status 404 returned error can't find the container with id b2f1dcc6808288c856424bb47b943f0a203cef578324483e6e628bb350134dcb
Jul 14 18:56:58.332957 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:58.332922    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9e94bd1-5c75-4816-ae29-022f5b8152ad" containerID="2df5d1058acc9e249c41615a3c6a301fee9b50b3eaa31392721cf2bbc414048a" exitCode=0
Jul 14 18:56:58.333137 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:58.333008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerDied","Data":"2df5d1058acc9e249c41615a3c6a301fee9b50b3eaa31392721cf2bbc414048a"}
Jul 14 18:56:58.333137 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:58.333072    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"b2f1dcc6808288c856424bb47b943f0a203cef578324483e6e628bb350134dcb"}
Jul 14 18:56:58.622487 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:58.622432    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a6350b-7bb0-4296-9319-3e96936407bb" path="/var/lib/kubelet/pods/03a6350b-7bb0-4296-9319-3e96936407bb/volumes"
Jul 14 18:56:59.341967 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.341931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"75acc89ace933b86d720d093baf6bcacfa9f56ec61d9d326068d63e5b80a7265"}
Jul 14 18:56:59.342352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.341997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"cb8f2302e876e8862c95b81405fa6becee3979f78283ae7fa29daf46eafc578b"}
Jul 14 18:56:59.342352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.342022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"c62c6e47455a2448f5c2bad1d3e77134974ae2d91f6a12dfde51231fccd85f6b"}
Jul 14 18:56:59.342352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.342050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"f9402015ddf6dd1552d66faa60c5bc3400e44ef52a645d3be7a1e52197f8f0fc"}
Jul 14 18:56:59.342352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.342074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"525ce2599fd81def37871536c5e590077e9b25cd1f9e899b38c8791648c1d930"}
Jul 14 18:56:59.342352 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.342095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f9e94bd1-5c75-4816-ae29-022f5b8152ad","Type":"ContainerStarted","Data":"f64e13ad57a8aa5aa4b1ee707fd95f5252e4f82ee93389ff4574569508b4e954"}
Jul 14 18:56:59.387512 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:56:59.387458    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.366905272 podStartE2EDuration="2.387439009s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.533737ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:59.38465811 +0000 UTC m=+163.257216342" watchObservedRunningTime="2026-07-14 18:56:59.387439009 +0000 UTC m=+163.259997227"
Jul 14 18:57:00.005728 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:00.005696    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:00.064552 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:00.064522    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:02.509200 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:02.509164    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:02.609937 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:02.609906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:02.676769 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:02.676740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:02.704688 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:02.704650    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:57:04.738136 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:04.738106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:04.838337 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:04.838290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:35.107941 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:35.107909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:35.185525 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:35.185494    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:40.986553 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:40.986519    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:41.085372 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:41.085340    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:57:57.704718 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:57.704678    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:57:57.720638 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:57.720610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:57:58.576454 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:57:58.576428    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 18:58:08.115656 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:08.115534    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:09.519746 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:09.519715    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:11.958168 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:11.958132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:12.142964 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:12.142930    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:16.726166 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:16.726131    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:17.567294 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:17.567264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:19.036752 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:19.036716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:19.189127 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:19.189097    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:21.005802 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:21.005767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:21.591084 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:21.591055    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:28.132862 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:28.132831    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:28.232956 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:28.232925    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:29.055261 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:29.055228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:30.748708 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.748669    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4h59g"]
Jul 14 18:58:30.751764 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.751748    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:30.754707 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.754675    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jgscj\""
Jul 14 18:58:30.754707 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.754675    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 18:58:30.755484 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.755470    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 18:58:30.762090 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.762069    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4h59g"]
Jul 14 18:58:30.817137 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.817093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7dh\" (UniqueName: \"kubernetes.io/projected/e4be0034-4ba6-4ac0-926b-60da9d5d0a9f-kube-api-access-kk7dh\") pod \"cert-manager-55b4d954d5-4h59g\" (UID: \"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f\") " pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:30.817311 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.817154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4be0034-4ba6-4ac0-926b-60da9d5d0a9f-bound-sa-token\") pod \"cert-manager-55b4d954d5-4h59g\" (UID: \"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f\") " pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:30.918075 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.918025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7dh\" (UniqueName: \"kubernetes.io/projected/e4be0034-4ba6-4ac0-926b-60da9d5d0a9f-kube-api-access-kk7dh\") pod \"cert-manager-55b4d954d5-4h59g\" (UID: \"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f\") " pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:30.918264 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.918108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4be0034-4ba6-4ac0-926b-60da9d5d0a9f-bound-sa-token\") pod \"cert-manager-55b4d954d5-4h59g\" (UID: \"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f\") " pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:30.930570 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.930535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e4be0034-4ba6-4ac0-926b-60da9d5d0a9f-bound-sa-token\") pod \"cert-manager-55b4d954d5-4h59g\" (UID: \"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f\") " pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:30.930701 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:30.930587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7dh\" (UniqueName: \"kubernetes.io/projected/e4be0034-4ba6-4ac0-926b-60da9d5d0a9f-kube-api-access-kk7dh\") pod \"cert-manager-55b4d954d5-4h59g\" (UID: \"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f\") " pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:31.071998 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:31.071916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4h59g"
Jul 14 18:58:31.204506 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:31.204481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4h59g"]
Jul 14 18:58:31.206786 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:58:31.206759    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4be0034_4ba6_4ac0_926b_60da9d5d0a9f.slice/crio-be5ed232537c5b5bc7c36e4d575d09055d29c98ad2a353c047fe6165e91b26ff WatchSource:0}: Error finding container be5ed232537c5b5bc7c36e4d575d09055d29c98ad2a353c047fe6165e91b26ff: Status 404 returned error can't find the container with id be5ed232537c5b5bc7c36e4d575d09055d29c98ad2a353c047fe6165e91b26ff
Jul 14 18:58:31.676450 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:31.676378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4h59g" event={"ID":"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f","Type":"ContainerStarted","Data":"be5ed232537c5b5bc7c36e4d575d09055d29c98ad2a353c047fe6165e91b26ff"}
Jul 14 18:58:34.545959 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:34.545926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:34.648976 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:34.648948    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:34.690590 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:34.690554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4h59g" event={"ID":"e4be0034-4ba6-4ac0-926b-60da9d5d0a9f","Type":"ContainerStarted","Data":"c4767fb3c703aee217977f83fa4fdc895cfc83a3494718dc2c1f6b6602f127b7"}
Jul 14 18:58:34.709859 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:34.709790    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4h59g" podStartSLOduration=1.985445597 podStartE2EDuration="4.70976686s" totalImagesPullingTime="2.724321263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:58:34.70782002 +0000 UTC m=+258.580378240" watchObservedRunningTime="2026-07-14 18:58:34.70976686 +0000 UTC m=+258.582325077"
Jul 14 18:58:36.308751 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:36.308716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:48.143613 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:48.143521    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:48.244792 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:48.244759    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:48.669716 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:48.669675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:48.881047 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:48.881006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:48.990542 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:48.990500    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:49.321806 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.321719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:49.412577 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.412535    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:49.475995 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.475954    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"]
Jul 14 18:58:49.480674 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.480653    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.485893 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.485869    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 18:58:49.486060 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.485902    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 18:58:49.486060 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.485938    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 18:58:49.486347 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.486329    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dxtk8\""
Jul 14 18:58:49.486784 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.486769    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 18:58:49.501965 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.501937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"]
Jul 14 18:58:49.574642 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.574546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-tmp\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.574642 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.574595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.574826 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.574645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-manifests\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.574826 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.574667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-kube-api-access-bfpn6\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.574826 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.574694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.624495 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.624458    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:49.675815 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.675781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-kube-api-access-bfpn6\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.675815 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.675819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.676027 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.675872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-tmp\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.676027 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.675897    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.676027 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.675947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-manifests\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.676387 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.676360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-manifests\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.678338 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.678309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-tmp\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.678452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.678389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.678452 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.678434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.689133 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.689107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899-kube-api-access-bfpn6\") pod \"opendatahub-operator-controller-manager-5ff67ccb56-w97qn\" (UID: \"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.790980 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.790934    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:49.938299 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.938262    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"]
Jul 14 18:58:49.942989 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:58:49.942954    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c907d3f_ecd0_4fa1_80ea_9c0ed58e8899.slice/crio-3bfc3a5f23f24398bc2cec55ebe030ce38451522a9078aa5f4e1291f4d5ecd99 WatchSource:0}: Error finding container 3bfc3a5f23f24398bc2cec55ebe030ce38451522a9078aa5f4e1291f4d5ecd99: Status 404 returned error can't find the container with id 3bfc3a5f23f24398bc2cec55ebe030ce38451522a9078aa5f4e1291f4d5ecd99
Jul 14 18:58:49.951759 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:49.951733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:50.078851 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:50.078810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:50.174947 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:50.174864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:50.753515 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:50.753477    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn" event={"ID":"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899","Type":"ContainerStarted","Data":"3bfc3a5f23f24398bc2cec55ebe030ce38451522a9078aa5f4e1291f4d5ecd99"}
Jul 14 18:58:53.123572 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:53.123547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:53.224515 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:53.224482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:53.769263 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:53.769225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn" event={"ID":"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899","Type":"ContainerStarted","Data":"ca2dbc89bb89e8e3c4171856d1b1b24289a64ade32047ea02a7cfaf914f87ea5"}
Jul 14 18:58:54.774503 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:54.774464    2562 generic.go:356] "Generic (PLEG): container finished" podID="4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899" containerID="ca2dbc89bb89e8e3c4171856d1b1b24289a64ade32047ea02a7cfaf914f87ea5" exitCode=0
Jul 14 18:58:54.774889 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:54.774525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn" event={"ID":"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899","Type":"ContainerDied","Data":"ca2dbc89bb89e8e3c4171856d1b1b24289a64ade32047ea02a7cfaf914f87ea5"}
Jul 14 18:58:55.780485 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:55.780451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn" event={"ID":"4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899","Type":"ContainerStarted","Data":"ed3ea0ad641bc43c175aeaeb6b27893cee3d1fcf87f81f21479511af1ab72ef8"}
Jul 14 18:58:55.780890 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:55.780551    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:58:55.810695 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:55.810641    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn" podStartSLOduration=2.2415123120000002 podStartE2EDuration="6.810620518s" totalImagesPullingTime="3.547997624s" totalInitContainerRuntime="1.021110582s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:58:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:58:55.80908542 +0000 UTC m=+279.681643635" watchObservedRunningTime="2026-07-14 18:58:55.810620518 +0000 UTC m=+279.683178739"
Jul 14 18:58:58.552617 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:58.552578    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:58:58.773513 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:58:58.773477    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:06.212970 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:06.212929    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:06.419028 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:06.418996    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:06.667592 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:06.667551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:06.765209 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:06.765173    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:06.786750 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:06.786723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff67ccb56-w97qn"
Jul 14 18:59:09.270983 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.270952    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"]
Jul 14 18:59:09.274262 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.274244    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.278813 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.278791    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 18:59:09.278813 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.278795    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 18:59:09.279125 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.279100    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 18:59:09.279962 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.279939    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rrmr9\""
Jul 14 18:59:09.281390 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.281374    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 18:59:09.281857 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.281844    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 18:59:09.292474 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.292450    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"]
Jul 14 18:59:09.330561 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.330522    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30604a39-56e0-44d2-97e1-34ab12984b23-metrics-cert\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.330753 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.330585    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30604a39-56e0-44d2-97e1-34ab12984b23-cert\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.330753 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.330631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30604a39-56e0-44d2-97e1-34ab12984b23-manager-config\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.330753 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.330655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmlrm\" (UniqueName: \"kubernetes.io/projected/30604a39-56e0-44d2-97e1-34ab12984b23-kube-api-access-xmlrm\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.431049 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.431013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30604a39-56e0-44d2-97e1-34ab12984b23-cert\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.431049 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.431063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30604a39-56e0-44d2-97e1-34ab12984b23-manager-config\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.431303 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.431190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmlrm\" (UniqueName: \"kubernetes.io/projected/30604a39-56e0-44d2-97e1-34ab12984b23-kube-api-access-xmlrm\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.431303 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.431267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30604a39-56e0-44d2-97e1-34ab12984b23-metrics-cert\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.431803 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.431780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30604a39-56e0-44d2-97e1-34ab12984b23-manager-config\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.433759 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.433735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30604a39-56e0-44d2-97e1-34ab12984b23-cert\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.433867 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.433784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30604a39-56e0-44d2-97e1-34ab12984b23-metrics-cert\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.455121 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.455092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmlrm\" (UniqueName: \"kubernetes.io/projected/30604a39-56e0-44d2-97e1-34ab12984b23-kube-api-access-xmlrm\") pod \"lws-controller-manager-6c74864f54-wndml\" (UID: \"30604a39-56e0-44d2-97e1-34ab12984b23\") " pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.584121 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.584030    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:09.736277 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.736245    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"]
Jul 14 18:59:09.739178 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:59:09.739152    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30604a39_56e0_44d2_97e1_34ab12984b23.slice/crio-bb93b8d716a8957f0bf83e4e5141247b608ca5f13273cb9e7ea1fba2c81ba4d4 WatchSource:0}: Error finding container bb93b8d716a8957f0bf83e4e5141247b608ca5f13273cb9e7ea1fba2c81ba4d4: Status 404 returned error can't find the container with id bb93b8d716a8957f0bf83e4e5141247b608ca5f13273cb9e7ea1fba2c81ba4d4
Jul 14 18:59:09.748040 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.748017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:09.833093 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.833058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml" event={"ID":"30604a39-56e0-44d2-97e1-34ab12984b23","Type":"ContainerStarted","Data":"bb93b8d716a8957f0bf83e4e5141247b608ca5f13273cb9e7ea1fba2c81ba4d4"}
Jul 14 18:59:09.879949 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:09.879858    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:10.509157 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.509116    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:10.626521 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.625699    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"]
Jul 14 18:59:10.629464 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.629436    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.632601 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.632572    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 18:59:10.633700 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.633679    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 18:59:10.634468 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.633914    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bq9wx\""
Jul 14 18:59:10.634468 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.634098    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 18:59:10.634468 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.634297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 18:59:10.640952 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.640909    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9818b6-905d-4c31-8558-f30e13e4aae4-tls-certs\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.641107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.640961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qp9n\" (UniqueName: \"kubernetes.io/projected/9f9818b6-905d-4c31-8558-f30e13e4aae4-kube-api-access-5qp9n\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.641107 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.641071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f9818b6-905d-4c31-8558-f30e13e4aae4-tmp\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.644662 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.644633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"]
Jul 14 18:59:10.728284 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.728225    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:10.741552 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.741513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f9818b6-905d-4c31-8558-f30e13e4aae4-tmp\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.741744 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.741617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9818b6-905d-4c31-8558-f30e13e4aae4-tls-certs\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.741744 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.741660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qp9n\" (UniqueName: \"kubernetes.io/projected/9f9818b6-905d-4c31-8558-f30e13e4aae4-kube-api-access-5qp9n\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.744438 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.744396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f9818b6-905d-4c31-8558-f30e13e4aae4-tmp\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.744640 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.744617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f9818b6-905d-4c31-8558-f30e13e4aae4-tls-certs\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.755703 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.755673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qp9n\" (UniqueName: \"kubernetes.io/projected/9f9818b6-905d-4c31-8558-f30e13e4aae4-kube-api-access-5qp9n\") pod \"kube-auth-proxy-7bd89b4b77-n87wj\" (UID: \"9f9818b6-905d-4c31-8558-f30e13e4aae4\") " pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:10.884342 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.884232    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:10.944166 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:10.944118    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"
Jul 14 18:59:11.035813 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:11.035776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:11.123812 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:11.123645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj"]
Jul 14 18:59:11.159863 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:11.159784    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:11.270519 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:11.270481    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:12.065802 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:59:12.065769    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9818b6_905d_4c31_8558_f30e13e4aae4.slice/crio-5c7bbc4ff1dcfa11ff255dc55b7810be68d1a785c229ac104e9c8f87e824f7a1 WatchSource:0}: Error finding container 5c7bbc4ff1dcfa11ff255dc55b7810be68d1a785c229ac104e9c8f87e824f7a1: Status 404 returned error can't find the container with id 5c7bbc4ff1dcfa11ff255dc55b7810be68d1a785c229ac104e9c8f87e824f7a1
Jul 14 18:59:12.206230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:12.206195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:12.550629 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:12.550599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:12.849177 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:12.849072    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml" event={"ID":"30604a39-56e0-44d2-97e1-34ab12984b23","Type":"ContainerStarted","Data":"b8fb092af463826ca071294ead057facad2ad115dc49d536cf084c8becdd6b48"}
Jul 14 18:59:12.849177 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:12.849142    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:12.850192 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:12.850165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj" event={"ID":"9f9818b6-905d-4c31-8558-f30e13e4aae4","Type":"ContainerStarted","Data":"5c7bbc4ff1dcfa11ff255dc55b7810be68d1a785c229ac104e9c8f87e824f7a1"}
Jul 14 18:59:13.112607 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:13.112524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:13.261800 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:13.261767    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:15.073156 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:15.073122    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:15.865055 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:15.864951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj" event={"ID":"9f9818b6-905d-4c31-8558-f30e13e4aae4","Type":"ContainerStarted","Data":"a49b25c25c6efc5f04865d8eb56e0e0a5ecc283db94715a251898528f91d2806"}
Jul 14 18:59:15.887202 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:15.887154    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml" podStartSLOduration=4.499678969 podStartE2EDuration="6.88713826s" totalImagesPullingTime="2.387459291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:12.876353627 +0000 UTC m=+296.748911846" watchObservedRunningTime="2026-07-14 18:59:15.88713826 +0000 UTC m=+299.759696481"
Jul 14 18:59:15.887359 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:15.887232    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bd89b4b77-n87wj" podStartSLOduration=2.342294198 podStartE2EDuration="5.887228128s" totalImagesPullingTime="3.54493393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:15.884875757 +0000 UTC m=+299.757433974" watchObservedRunningTime="2026-07-14 18:59:15.887228128 +0000 UTC m=+299.759786347"
Jul 14 18:59:16.507122 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:16.507098    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 18:59:18.081052 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:18.081015    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:18.236163 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:18.236132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:18.594764 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:18.594730    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:18.909692 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:18.909596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:18.990982 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:18.990951    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:22.438149 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.438114    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"]
Jul 14 18:59:22.441545 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.441524    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.446832 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.446810    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 14 18:59:22.447323 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.447301    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gf48s\""
Jul 14 18:59:22.459388 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.459357    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"]
Jul 14 18:59:22.541082 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.541045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35434038-8fd6-4337-ab44-5b975cba860a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc4h8\" (UID: \"35434038-8fd6-4337-ab44-5b975cba860a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.541257 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.541100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9bb\" (UniqueName: \"kubernetes.io/projected/35434038-8fd6-4337-ab44-5b975cba860a-kube-api-access-7x9bb\") pod \"llmisvc-controller-manager-777c6bd7d5-tc4h8\" (UID: \"35434038-8fd6-4337-ab44-5b975cba860a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.641736 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.641699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35434038-8fd6-4337-ab44-5b975cba860a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc4h8\" (UID: \"35434038-8fd6-4337-ab44-5b975cba860a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.641927 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.641750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9bb\" (UniqueName: \"kubernetes.io/projected/35434038-8fd6-4337-ab44-5b975cba860a-kube-api-access-7x9bb\") pod \"llmisvc-controller-manager-777c6bd7d5-tc4h8\" (UID: \"35434038-8fd6-4337-ab44-5b975cba860a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.644230 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.644209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/35434038-8fd6-4337-ab44-5b975cba860a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tc4h8\" (UID: \"35434038-8fd6-4337-ab44-5b975cba860a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.660761 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.660735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9bb\" (UniqueName: \"kubernetes.io/projected/35434038-8fd6-4337-ab44-5b975cba860a-kube-api-access-7x9bb\") pod \"llmisvc-controller-manager-777c6bd7d5-tc4h8\" (UID: \"35434038-8fd6-4337-ab44-5b975cba860a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.714585 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.714541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:22.751119 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.751082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:22.809838 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.809801    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:22.909256 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.909226    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"]
Jul 14 18:59:22.921557 ip-10-0-132-123 kubenswrapper[2562]: W0714 18:59:22.921528    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35434038_8fd6_4337_ab44_5b975cba860a.slice/crio-b6ea57c9b35bfc687bc6cb52e7195a27e8a002b0215b70c920b432164436ce57 WatchSource:0}: Error finding container b6ea57c9b35bfc687bc6cb52e7195a27e8a002b0215b70c920b432164436ce57: Status 404 returned error can't find the container with id b6ea57c9b35bfc687bc6cb52e7195a27e8a002b0215b70c920b432164436ce57
Jul 14 18:59:22.923258 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:22.923238    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 18:59:23.859788 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:23.859756    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c74864f54-wndml"
Jul 14 18:59:23.898283 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:23.898247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8" event={"ID":"35434038-8fd6-4337-ab44-5b975cba860a","Type":"ContainerStarted","Data":"b6ea57c9b35bfc687bc6cb52e7195a27e8a002b0215b70c920b432164436ce57"}
Jul 14 18:59:24.179927 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:24.179828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:24.286666 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:24.286627    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:25.637176 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:25.637140    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:25.765636 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:25.765594    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:27.791759 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:27.791723    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:27.916307 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:27.916272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8" event={"ID":"35434038-8fd6-4337-ab44-5b975cba860a","Type":"ContainerStarted","Data":"5229259a7d2ade627ca079b0f54e3492c36bbc64d4e69d360c0324163ac1d390"}
Jul 14 18:59:27.916495 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:27.916337    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 18:59:27.962150 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:27.962104    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8" podStartSLOduration=1.940123239 podStartE2EDuration="5.962090087s" totalImagesPullingTime="4.021966848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:27.960592066 +0000 UTC m=+311.833150286" watchObservedRunningTime="2026-07-14 18:59:27.962090087 +0000 UTC m=+311.834648306"
Jul 14 18:59:34.736084 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:34.736052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:34.834843 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:34.834808    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 18:59:58.923089 ip-10-0-132-123 kubenswrapper[2562]: I0714 18:59:58.923057    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tc4h8"
Jul 14 19:00:00.488645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:00.488610    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:11.198838 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:11.198748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:11.296301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:11.296260    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:11.432843 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:11.432807    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:43.017155 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:43.017120    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:44.446193 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:44.446157    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:45.329582 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:45.329554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:46.497906 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:46.497873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:49.061421 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:49.061371    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:51.525715 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:51.525682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:51.867905 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:51.867824    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:52.723631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:52.723596    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:54.538195 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:54.538155    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:57.491010 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:57.490976    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:00:57.600597 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:00:57.600563    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:02.857354 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:02.857317    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:03.060152 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:03.060117    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:05.803775 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:05.803740    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:05.917100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:05.917073    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:08.059681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:08.059648    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:08.178213 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:08.178181    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:10.096933 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:10.096905    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:12.058430 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:12.058373    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:12.160363 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:12.160330    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:18.007332 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:18.007293    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:18.121095 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:18.121051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:19.545440 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.545371    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"]
Jul 14 19:01:19.550311 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.550285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.557943 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.557922    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 19:01:19.558167 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.558154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wbx4n\""
Jul 14 19:01:19.558236 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.558156    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 19:01:19.575793 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.575767    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"]
Jul 14 19:01:19.656614 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.656579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee81bd74-d21d-4ecd-a49d-c3eed90df472-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5qtw6\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.656799 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.656645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgzz\" (UniqueName: \"kubernetes.io/projected/ee81bd74-d21d-4ecd-a49d-c3eed90df472-kube-api-access-rmgzz\") pod \"kuadrant-operator-controller-manager-86878c769f-5qtw6\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.758114 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.758077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee81bd74-d21d-4ecd-a49d-c3eed90df472-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5qtw6\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.758276 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.758143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgzz\" (UniqueName: \"kubernetes.io/projected/ee81bd74-d21d-4ecd-a49d-c3eed90df472-kube-api-access-rmgzz\") pod \"kuadrant-operator-controller-manager-86878c769f-5qtw6\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.758516 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.758490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee81bd74-d21d-4ecd-a49d-c3eed90df472-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5qtw6\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.768624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.768603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgzz\" (UniqueName: \"kubernetes.io/projected/ee81bd74-d21d-4ecd-a49d-c3eed90df472-kube-api-access-rmgzz\") pod \"kuadrant-operator-controller-manager-86878c769f-5qtw6\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:19.860137 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:19.860042    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:20.003301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:20.003270    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"]
Jul 14 19:01:20.006749 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:01:20.006722    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee81bd74_d21d_4ecd_a49d_c3eed90df472.slice/crio-cdc195b11f9780b35cf9dd2eda8e0f79b4480f903464667d9700b8a75bc30c5f WatchSource:0}: Error finding container cdc195b11f9780b35cf9dd2eda8e0f79b4480f903464667d9700b8a75bc30c5f: Status 404 returned error can't find the container with id cdc195b11f9780b35cf9dd2eda8e0f79b4480f903464667d9700b8a75bc30c5f
Jul 14 19:01:20.378571 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:20.378539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6" event={"ID":"ee81bd74-d21d-4ecd-a49d-c3eed90df472","Type":"ContainerStarted","Data":"cdc195b11f9780b35cf9dd2eda8e0f79b4480f903464667d9700b8a75bc30c5f"}
Jul 14 19:01:23.377149 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:23.377111    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:23.468728 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:23.468702    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:23.848656 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:23.848622    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:25.401652 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:25.401565    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6" event={"ID":"ee81bd74-d21d-4ecd-a49d-c3eed90df472","Type":"ContainerStarted","Data":"06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c"}
Jul 14 19:01:25.402056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:25.401694    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:25.431862 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:25.431814    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6" podStartSLOduration=1.353165733 podStartE2EDuration="6.43179916s" totalImagesPullingTime="5.078633427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:25.429742682 +0000 UTC m=+429.302300903" watchObservedRunningTime="2026-07-14 19:01:25.43179916 +0000 UTC m=+429.304357426"
Jul 14 19:01:25.558881 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:25.558847    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:27.157543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:27.157512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:27.259746 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:27.259675    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:30.777695 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:30.777655    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:30.880260 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:30.880226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:36.408594 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:36.408555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:01:53.163433 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:53.163339    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:53.663070 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:53.663040    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:56.407750 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:56.407720    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:56.549056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:56.549020    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:56.648847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:56.648820    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:57.308790 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.308758    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-529rz"]
Jul 14 19:01:57.312855 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.312835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:01:57.315599 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.315570    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d4jcb\""
Jul 14 19:01:57.320578 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.320557    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-529rz"]
Jul 14 19:01:57.348772 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.348713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:57.380105 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.380075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbd6s\" (UniqueName: \"kubernetes.io/projected/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8-kube-api-access-bbd6s\") pod \"authorino-58cd955675-529rz\" (UID: \"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8\") " pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:01:57.423572 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.423539    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:57.481485 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.481448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbd6s\" (UniqueName: \"kubernetes.io/projected/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8-kube-api-access-bbd6s\") pod \"authorino-58cd955675-529rz\" (UID: \"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8\") " pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:01:57.497869 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.497834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbd6s\" (UniqueName: \"kubernetes.io/projected/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8-kube-api-access-bbd6s\") pod \"authorino-58cd955675-529rz\" (UID: \"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8\") " pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:01:57.530421 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.530382    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-dz8gs"]
Jul 14 19:01:57.532863 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.532846    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:01:57.555124 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.555092    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dz8gs"]
Jul 14 19:01:57.623468 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.623361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:01:57.682813 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.682781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgclq\" (UniqueName: \"kubernetes.io/projected/9106aa0f-555d-491d-8f35-c38f81e4c040-kube-api-access-zgclq\") pod \"authorino-75d556bd77-dz8gs\" (UID: \"9106aa0f-555d-491d-8f35-c38f81e4c040\") " pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:01:57.757079 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.756971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-529rz"]
Jul 14 19:01:57.759510 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:01:57.759476    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9c0af3c_44e7_48bb_b72f_b7f1eb37f3b8.slice/crio-15382cbe3de92d7afdf242188450a075ec6033c8aa6c1a2b84e0b47da86c00d8 WatchSource:0}: Error finding container 15382cbe3de92d7afdf242188450a075ec6033c8aa6c1a2b84e0b47da86c00d8: Status 404 returned error can't find the container with id 15382cbe3de92d7afdf242188450a075ec6033c8aa6c1a2b84e0b47da86c00d8
Jul 14 19:01:57.784111 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.784069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgclq\" (UniqueName: \"kubernetes.io/projected/9106aa0f-555d-491d-8f35-c38f81e4c040-kube-api-access-zgclq\") pod \"authorino-75d556bd77-dz8gs\" (UID: \"9106aa0f-555d-491d-8f35-c38f81e4c040\") " pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:01:57.800549 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.800524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgclq\" (UniqueName: \"kubernetes.io/projected/9106aa0f-555d-491d-8f35-c38f81e4c040-kube-api-access-zgclq\") pod \"authorino-75d556bd77-dz8gs\" (UID: \"9106aa0f-555d-491d-8f35-c38f81e4c040\") " pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:01:57.847740 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.847703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:01:57.979093 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:57.979026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dz8gs"]
Jul 14 19:01:57.981582 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:01:57.981554    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9106aa0f_555d_491d_8f35_c38f81e4c040.slice/crio-66a876bdb7412d9c5d9758103950c64e773bf7c180b1e011fc24431b2c76e775 WatchSource:0}: Error finding container 66a876bdb7412d9c5d9758103950c64e773bf7c180b1e011fc24431b2c76e775: Status 404 returned error can't find the container with id 66a876bdb7412d9c5d9758103950c64e773bf7c180b1e011fc24431b2c76e775
Jul 14 19:01:58.551324 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:58.551289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-529rz" event={"ID":"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8","Type":"ContainerStarted","Data":"15382cbe3de92d7afdf242188450a075ec6033c8aa6c1a2b84e0b47da86c00d8"}
Jul 14 19:01:58.552474 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:58.552438    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dz8gs" event={"ID":"9106aa0f-555d-491d-8f35-c38f81e4c040","Type":"ContainerStarted","Data":"66a876bdb7412d9c5d9758103950c64e773bf7c180b1e011fc24431b2c76e775"}
Jul 14 19:01:59.491678 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:59.491640    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:01:59.563887 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:01:59.563852    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:01.568357 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:01.568319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-529rz" event={"ID":"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8","Type":"ContainerStarted","Data":"3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd"}
Jul 14 19:02:01.569683 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:01.569654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dz8gs" event={"ID":"9106aa0f-555d-491d-8f35-c38f81e4c040","Type":"ContainerStarted","Data":"409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2"}
Jul 14 19:02:01.589716 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:01.589656    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-529rz" podStartSLOduration=1.208907525 podStartE2EDuration="4.589641993s" totalImagesPullingTime="3.380734468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:01.588655471 +0000 UTC m=+465.461213693" watchObservedRunningTime="2026-07-14 19:02:01.589641993 +0000 UTC m=+465.462200213"
Jul 14 19:02:01.611847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:01.611801    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-dz8gs" podStartSLOduration=1.455468994 podStartE2EDuration="4.61178216s" totalImagesPullingTime="3.156313166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:01.610821122 +0000 UTC m=+465.483379343" watchObservedRunningTime="2026-07-14 19:02:01.61178216 +0000 UTC m=+465.484340380"
Jul 14 19:02:01.644348 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:01.644312    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-529rz"]
Jul 14 19:02:03.584735 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:03.584695    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-529rz" podUID="c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" containerName="authorino" containerID="cri-o://3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd" gracePeriod=30
Jul 14 19:02:03.831827 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:03.831804    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:02:03.939309 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:03.939230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbd6s\" (UniqueName: \"kubernetes.io/projected/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8-kube-api-access-bbd6s\") pod \"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8\" (UID: \"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8\") "
Jul 14 19:02:03.941559 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:03.941522    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8-kube-api-access-bbd6s" (OuterVolumeSpecName: "kube-api-access-bbd6s") pod "c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" (UID: "c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8"). InnerVolumeSpecName "kube-api-access-bbd6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:02:04.040249 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.040209    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbd6s\" (UniqueName: \"kubernetes.io/projected/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8-kube-api-access-bbd6s\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:02:04.590319 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.590286    2562 generic.go:356] "Generic (PLEG): container finished" podID="c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" containerID="3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd" exitCode=0
Jul 14 19:02:04.590319 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.590325    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-529rz"
Jul 14 19:02:04.590826 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.590383    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-529rz" event={"ID":"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8","Type":"ContainerDied","Data":"3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd"}
Jul 14 19:02:04.590826 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.590443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-529rz" event={"ID":"c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8","Type":"ContainerDied","Data":"15382cbe3de92d7afdf242188450a075ec6033c8aa6c1a2b84e0b47da86c00d8"}
Jul 14 19:02:04.590826 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.590464    2562 scope.go:117] "RemoveContainer" containerID="3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd"
Jul 14 19:02:04.598898 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.598880    2562 scope.go:117] "RemoveContainer" containerID="3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd"
Jul 14 19:02:04.599174 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:02:04.599155    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd\": container with ID starting with 3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd not found: ID does not exist" containerID="3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd"
Jul 14 19:02:04.599237 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.599186    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd"} err="failed to get container status \"3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd\": rpc error: code = NotFound desc = could not find container \"3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd\": container with ID starting with 3ea0067f5e6990d73828fb547d91d4352ba321b16e4122bb29c095786a3897fd not found: ID does not exist"
Jul 14 19:02:04.612431 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.612387    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-529rz"]
Jul 14 19:02:04.620933 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:04.620904    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-529rz"]
Jul 14 19:02:06.621077 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:06.621042    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" path="/var/lib/kubelet/pods/c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8/volumes"
Jul 14 19:02:16.358173 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:16.358139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:16.464586 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:16.464553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:16.917198 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:16.917163    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:19.741461 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:19.741420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:19.844981 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:19.844947    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:27.228168 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:27.228128    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:29.266472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.266436    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k4dzj"]
Jul 14 19:02:29.266854 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.266766    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" containerName="authorino"
Jul 14 19:02:29.266854 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.266777    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" containerName="authorino"
Jul 14 19:02:29.266854 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.266825    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c0af3c-44e7-48bb-b72f-b7f1eb37f3b8" containerName="authorino"
Jul 14 19:02:29.270938 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.270919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:29.280836 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.280811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k4dzj"]
Jul 14 19:02:29.362202 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.362171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lprf\" (UniqueName: \"kubernetes.io/projected/c73ace37-b86b-4a89-9f15-3c53f19b5d4b-kube-api-access-8lprf\") pod \"authorino-5d4b67c446-k4dzj\" (UID: \"c73ace37-b86b-4a89-9f15-3c53f19b5d4b\") " pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:29.463292 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.463251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lprf\" (UniqueName: \"kubernetes.io/projected/c73ace37-b86b-4a89-9f15-3c53f19b5d4b-kube-api-access-8lprf\") pod \"authorino-5d4b67c446-k4dzj\" (UID: \"c73ace37-b86b-4a89-9f15-3c53f19b5d4b\") " pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:29.480421 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.480377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lprf\" (UniqueName: \"kubernetes.io/projected/c73ace37-b86b-4a89-9f15-3c53f19b5d4b-kube-api-access-8lprf\") pod \"authorino-5d4b67c446-k4dzj\" (UID: \"c73ace37-b86b-4a89-9f15-3c53f19b5d4b\") " pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:29.533947 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.533869    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k4dzj"]
Jul 14 19:02:29.534124 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.534113    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:29.582130 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.582096    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b545d56cd-xqmpt"]
Jul 14 19:02:29.585243 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.585221    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:29.604941 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.604907    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b545d56cd-xqmpt"]
Jul 14 19:02:29.664970 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.664928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmml\" (UniqueName: \"kubernetes.io/projected/03de250e-9d31-4b1f-b607-1e958954d754-kube-api-access-6xmml\") pod \"authorino-7b545d56cd-xqmpt\" (UID: \"03de250e-9d31-4b1f-b607-1e958954d754\") " pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:29.673188 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.673166    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k4dzj"]
Jul 14 19:02:29.675723 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:02:29.675684    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc73ace37_b86b_4a89_9f15_3c53f19b5d4b.slice/crio-e1e12c29da4612c833133c32294e710e6bde9806aaaf352b534f820e47533614 WatchSource:0}: Error finding container e1e12c29da4612c833133c32294e710e6bde9806aaaf352b534f820e47533614: Status 404 returned error can't find the container with id e1e12c29da4612c833133c32294e710e6bde9806aaaf352b534f820e47533614
Jul 14 19:02:29.695305 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.695268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k4dzj" event={"ID":"c73ace37-b86b-4a89-9f15-3c53f19b5d4b","Type":"ContainerStarted","Data":"e1e12c29da4612c833133c32294e710e6bde9806aaaf352b534f820e47533614"}
Jul 14 19:02:29.766113 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.766076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmml\" (UniqueName: \"kubernetes.io/projected/03de250e-9d31-4b1f-b607-1e958954d754-kube-api-access-6xmml\") pod \"authorino-7b545d56cd-xqmpt\" (UID: \"03de250e-9d31-4b1f-b607-1e958954d754\") " pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:29.777663 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.777634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmml\" (UniqueName: \"kubernetes.io/projected/03de250e-9d31-4b1f-b607-1e958954d754-kube-api-access-6xmml\") pod \"authorino-7b545d56cd-xqmpt\" (UID: \"03de250e-9d31-4b1f-b607-1e958954d754\") " pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:29.892543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.892459    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b545d56cd-xqmpt"]
Jul 14 19:02:29.892735 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.892724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:29.944678 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.944650    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77f494c9cd-6kc7f"]
Jul 14 19:02:29.947713 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.947685    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:29.951707 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.951436    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 14 19:02:29.959950 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:29.959925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77f494c9cd-6kc7f"]
Jul 14 19:02:30.027052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.026848    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b545d56cd-xqmpt"]
Jul 14 19:02:30.036620 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:02:30.036576    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03de250e_9d31_4b1f_b607_1e958954d754.slice/crio-9550b2f8f2f4d579467ea11d86c31cb014cb547866740baa0048a6a572cb685e WatchSource:0}: Error finding container 9550b2f8f2f4d579467ea11d86c31cb014cb547866740baa0048a6a572cb685e: Status 404 returned error can't find the container with id 9550b2f8f2f4d579467ea11d86c31cb014cb547866740baa0048a6a572cb685e
Jul 14 19:02:30.069258 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.069232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncjf\" (UniqueName: \"kubernetes.io/projected/93b9b4e5-621c-4771-a2c2-74e732715ea0-kube-api-access-gncjf\") pod \"authorino-77f494c9cd-6kc7f\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") " pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.069396 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.069363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93b9b4e5-621c-4771-a2c2-74e732715ea0-tls-cert\") pod \"authorino-77f494c9cd-6kc7f\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") " pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.170679 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.170595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncjf\" (UniqueName: \"kubernetes.io/projected/93b9b4e5-621c-4771-a2c2-74e732715ea0-kube-api-access-gncjf\") pod \"authorino-77f494c9cd-6kc7f\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") " pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.170679 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.170673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93b9b4e5-621c-4771-a2c2-74e732715ea0-tls-cert\") pod \"authorino-77f494c9cd-6kc7f\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") " pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.173147 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.173129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93b9b4e5-621c-4771-a2c2-74e732715ea0-tls-cert\") pod \"authorino-77f494c9cd-6kc7f\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") " pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.180473 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.180453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncjf\" (UniqueName: \"kubernetes.io/projected/93b9b4e5-621c-4771-a2c2-74e732715ea0-kube-api-access-gncjf\") pod \"authorino-77f494c9cd-6kc7f\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") " pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.260296 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.260253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:02:30.285903 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.285864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:30.367960 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.367924    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:30.413686 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.413660    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77f494c9cd-6kc7f"]
Jul 14 19:02:30.415197 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:02:30.415167    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93b9b4e5_621c_4771_a2c2_74e732715ea0.slice/crio-a090be2de6ab10d229b1a095b46db602b8ef2603c4d869a89ff9558e43cd0847 WatchSource:0}: Error finding container a090be2de6ab10d229b1a095b46db602b8ef2603c4d869a89ff9558e43cd0847: Status 404 returned error can't find the container with id a090be2de6ab10d229b1a095b46db602b8ef2603c4d869a89ff9558e43cd0847
Jul 14 19:02:30.701242 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.701146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k4dzj" event={"ID":"c73ace37-b86b-4a89-9f15-3c53f19b5d4b","Type":"ContainerStarted","Data":"01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3"}
Jul 14 19:02:30.701242 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.701192    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-k4dzj" podUID="c73ace37-b86b-4a89-9f15-3c53f19b5d4b" containerName="authorino" containerID="cri-o://01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3" gracePeriod=30
Jul 14 19:02:30.702555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.702531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f494c9cd-6kc7f" event={"ID":"93b9b4e5-621c-4771-a2c2-74e732715ea0","Type":"ContainerStarted","Data":"a090be2de6ab10d229b1a095b46db602b8ef2603c4d869a89ff9558e43cd0847"}
Jul 14 19:02:30.703849 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.703830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b545d56cd-xqmpt" event={"ID":"03de250e-9d31-4b1f-b607-1e958954d754","Type":"ContainerStarted","Data":"e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8"}
Jul 14 19:02:30.703937 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.703857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b545d56cd-xqmpt" event={"ID":"03de250e-9d31-4b1f-b607-1e958954d754","Type":"ContainerStarted","Data":"9550b2f8f2f4d579467ea11d86c31cb014cb547866740baa0048a6a572cb685e"}
Jul 14 19:02:30.703937 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.703915    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7b545d56cd-xqmpt" podUID="03de250e-9d31-4b1f-b607-1e958954d754" containerName="authorino" containerID="cri-o://e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8" gracePeriod=30
Jul 14 19:02:30.727801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.727753    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-k4dzj" podStartSLOduration=1.402900628 podStartE2EDuration="1.727735616s" totalImagesPullingTime="324.834988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:30.725878149 +0000 UTC m=+494.598436369" watchObservedRunningTime="2026-07-14 19:02:30.727735616 +0000 UTC m=+494.600293835"
Jul 14 19:02:30.751339 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.751291    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b545d56cd-xqmpt" podStartSLOduration=1.355976909 podStartE2EDuration="1.751275319s" totalImagesPullingTime="395.29841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:30.750044307 +0000 UTC m=+494.622602527" watchObservedRunningTime="2026-07-14 19:02:30.751275319 +0000 UTC m=+494.623833538"
Jul 14 19:02:30.962160 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.962136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:30.971017 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:30.970993    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:31.080483 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.080437    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lprf\" (UniqueName: \"kubernetes.io/projected/c73ace37-b86b-4a89-9f15-3c53f19b5d4b-kube-api-access-8lprf\") pod \"c73ace37-b86b-4a89-9f15-3c53f19b5d4b\" (UID: \"c73ace37-b86b-4a89-9f15-3c53f19b5d4b\") "
Jul 14 19:02:31.080645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.080559    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xmml\" (UniqueName: \"kubernetes.io/projected/03de250e-9d31-4b1f-b607-1e958954d754-kube-api-access-6xmml\") pod \"03de250e-9d31-4b1f-b607-1e958954d754\" (UID: \"03de250e-9d31-4b1f-b607-1e958954d754\") "
Jul 14 19:02:31.082578 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.082547    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73ace37-b86b-4a89-9f15-3c53f19b5d4b-kube-api-access-8lprf" (OuterVolumeSpecName: "kube-api-access-8lprf") pod "c73ace37-b86b-4a89-9f15-3c53f19b5d4b" (UID: "c73ace37-b86b-4a89-9f15-3c53f19b5d4b"). InnerVolumeSpecName "kube-api-access-8lprf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:02:31.082695 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.082633    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03de250e-9d31-4b1f-b607-1e958954d754-kube-api-access-6xmml" (OuterVolumeSpecName: "kube-api-access-6xmml") pod "03de250e-9d31-4b1f-b607-1e958954d754" (UID: "03de250e-9d31-4b1f-b607-1e958954d754"). InnerVolumeSpecName "kube-api-access-6xmml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:02:31.181275 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.181238    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmml\" (UniqueName: \"kubernetes.io/projected/03de250e-9d31-4b1f-b607-1e958954d754-kube-api-access-6xmml\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:02:31.181275 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.181268    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lprf\" (UniqueName: \"kubernetes.io/projected/c73ace37-b86b-4a89-9f15-3c53f19b5d4b-kube-api-access-8lprf\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:02:31.709616 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.709579    2562 generic.go:356] "Generic (PLEG): container finished" podID="03de250e-9d31-4b1f-b607-1e958954d754" containerID="e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8" exitCode=0
Jul 14 19:02:31.710052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.709622    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b545d56cd-xqmpt"
Jul 14 19:02:31.710052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.709672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b545d56cd-xqmpt" event={"ID":"03de250e-9d31-4b1f-b607-1e958954d754","Type":"ContainerDied","Data":"e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8"}
Jul 14 19:02:31.710052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.709725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b545d56cd-xqmpt" event={"ID":"03de250e-9d31-4b1f-b607-1e958954d754","Type":"ContainerDied","Data":"9550b2f8f2f4d579467ea11d86c31cb014cb547866740baa0048a6a572cb685e"}
Jul 14 19:02:31.710052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.709753    2562 scope.go:117] "RemoveContainer" containerID="e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8"
Jul 14 19:02:31.717267 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.717239    2562 generic.go:356] "Generic (PLEG): container finished" podID="c73ace37-b86b-4a89-9f15-3c53f19b5d4b" containerID="01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3" exitCode=0
Jul 14 19:02:31.717431 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.717296    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-k4dzj"
Jul 14 19:02:31.717431 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.717316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k4dzj" event={"ID":"c73ace37-b86b-4a89-9f15-3c53f19b5d4b","Type":"ContainerDied","Data":"01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3"}
Jul 14 19:02:31.717431 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.717356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-k4dzj" event={"ID":"c73ace37-b86b-4a89-9f15-3c53f19b5d4b","Type":"ContainerDied","Data":"e1e12c29da4612c833133c32294e710e6bde9806aaaf352b534f820e47533614"}
Jul 14 19:02:31.719539 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.719500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f494c9cd-6kc7f" event={"ID":"93b9b4e5-621c-4771-a2c2-74e732715ea0","Type":"ContainerStarted","Data":"a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86"}
Jul 14 19:02:31.726273 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.726168    2562 scope.go:117] "RemoveContainer" containerID="e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8"
Jul 14 19:02:31.726546 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:02:31.726509    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8\": container with ID starting with e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8 not found: ID does not exist" containerID="e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8"
Jul 14 19:02:31.726633 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.726582    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8"} err="failed to get container status \"e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8\": rpc error: code = NotFound desc = could not find container \"e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8\": container with ID starting with e4d7ffdd9c732da07e58dd2b5412c89eee266552663491a89782d84424a081a8 not found: ID does not exist"
Jul 14 19:02:31.726633 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.726612    2562 scope.go:117] "RemoveContainer" containerID="01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3"
Jul 14 19:02:31.734515 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.734491    2562 scope.go:117] "RemoveContainer" containerID="01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3"
Jul 14 19:02:31.734783 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:02:31.734765    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3\": container with ID starting with 01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3 not found: ID does not exist" containerID="01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3"
Jul 14 19:02:31.734851 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.734792    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3"} err="failed to get container status \"01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3\": rpc error: code = NotFound desc = could not find container \"01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3\": container with ID starting with 01ed9bc16eadf5bafd9b7236edaffdc3707a2850410110a021b8cfef295bf6f3 not found: ID does not exist"
Jul 14 19:02:31.741674 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.741617    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b545d56cd-xqmpt"]
Jul 14 19:02:31.748487 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.748466    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b545d56cd-xqmpt"]
Jul 14 19:02:31.772733 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.772677    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77f494c9cd-6kc7f" podStartSLOduration=2.331778719 podStartE2EDuration="2.772659362s" totalImagesPullingTime="440.880643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:31.77013239 +0000 UTC m=+495.642690610" watchObservedRunningTime="2026-07-14 19:02:31.772659362 +0000 UTC m=+495.645217582"
Jul 14 19:02:31.806335 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.806305    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k4dzj"]
Jul 14 19:02:31.816799 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.816764    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-k4dzj"]
Jul 14 19:02:31.818093 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.818068    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dz8gs"]
Jul 14 19:02:31.818270 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:31.818249    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-dz8gs" podUID="9106aa0f-555d-491d-8f35-c38f81e4c040" containerName="authorino" containerID="cri-o://409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2" gracePeriod=30
Jul 14 19:02:32.064806 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.064782    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:02:32.191287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.191250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgclq\" (UniqueName: \"kubernetes.io/projected/9106aa0f-555d-491d-8f35-c38f81e4c040-kube-api-access-zgclq\") pod \"9106aa0f-555d-491d-8f35-c38f81e4c040\" (UID: \"9106aa0f-555d-491d-8f35-c38f81e4c040\") "
Jul 14 19:02:32.193558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.193520    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9106aa0f-555d-491d-8f35-c38f81e4c040-kube-api-access-zgclq" (OuterVolumeSpecName: "kube-api-access-zgclq") pod "9106aa0f-555d-491d-8f35-c38f81e4c040" (UID: "9106aa0f-555d-491d-8f35-c38f81e4c040"). InnerVolumeSpecName "kube-api-access-zgclq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:02:32.292148 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.292099    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgclq\" (UniqueName: \"kubernetes.io/projected/9106aa0f-555d-491d-8f35-c38f81e4c040-kube-api-access-zgclq\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:02:32.621326 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.621284    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03de250e-9d31-4b1f-b607-1e958954d754" path="/var/lib/kubelet/pods/03de250e-9d31-4b1f-b607-1e958954d754/volumes"
Jul 14 19:02:32.621733 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.621705    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73ace37-b86b-4a89-9f15-3c53f19b5d4b" path="/var/lib/kubelet/pods/c73ace37-b86b-4a89-9f15-3c53f19b5d4b/volumes"
Jul 14 19:02:32.725847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.725809    2562 generic.go:356] "Generic (PLEG): container finished" podID="9106aa0f-555d-491d-8f35-c38f81e4c040" containerID="409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2" exitCode=0
Jul 14 19:02:32.725847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.725849    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dz8gs"
Jul 14 19:02:32.726362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.725884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dz8gs" event={"ID":"9106aa0f-555d-491d-8f35-c38f81e4c040","Type":"ContainerDied","Data":"409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2"}
Jul 14 19:02:32.726362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.725927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dz8gs" event={"ID":"9106aa0f-555d-491d-8f35-c38f81e4c040","Type":"ContainerDied","Data":"66a876bdb7412d9c5d9758103950c64e773bf7c180b1e011fc24431b2c76e775"}
Jul 14 19:02:32.726362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.725988    2562 scope.go:117] "RemoveContainer" containerID="409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2"
Jul 14 19:02:32.734105 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.734087    2562 scope.go:117] "RemoveContainer" containerID="409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2"
Jul 14 19:02:32.734385 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:02:32.734366    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2\": container with ID starting with 409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2 not found: ID does not exist" containerID="409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2"
Jul 14 19:02:32.734456 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.734393    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2"} err="failed to get container status \"409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2\": rpc error: code = NotFound desc = could not find container \"409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2\": container with ID starting with 409f016e7e41f1e18f050238c2db843731779124e849bdc2fa3e085aaf1d7de2 not found: ID does not exist"
Jul 14 19:02:32.747028 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.746999    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dz8gs"]
Jul 14 19:02:32.763008 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:32.762983    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dz8gs"]
Jul 14 19:02:34.622108 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:34.622072    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9106aa0f-555d-491d-8f35-c38f81e4c040" path="/var/lib/kubelet/pods/9106aa0f-555d-491d-8f35-c38f81e4c040/volumes"
Jul 14 19:02:37.158814 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:37.158777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:37.254638 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:37.254599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:37.351809 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:37.351777    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:37.595078 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:37.595048    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:40.367765 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:40.367729    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:02:40.505595 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:02:40.505560    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:14.122517 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:14.122426    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:14.246178 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:14.246145    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:14.639811 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:14.639773    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:15.157283 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:15.157250    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:15.249737 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:15.249705    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:15.369025 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:15.368987    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:25.305278 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:25.305244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:27.965652 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:27.965619    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:30.230388 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:30.230352    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:30.331636 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:30.331601    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:31.601512 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:31.601482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:33.179593 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:33.179553    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:44.382900 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.382867    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"]
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383188    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03de250e-9d31-4b1f-b607-1e958954d754" containerName="authorino"
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383198    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="03de250e-9d31-4b1f-b607-1e958954d754" containerName="authorino"
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383207    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73ace37-b86b-4a89-9f15-3c53f19b5d4b" containerName="authorino"
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73ace37-b86b-4a89-9f15-3c53f19b5d4b" containerName="authorino"
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383226    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9106aa0f-555d-491d-8f35-c38f81e4c040" containerName="authorino"
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383231    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9106aa0f-555d-491d-8f35-c38f81e4c040" containerName="authorino"
Jul 14 19:03:44.383274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383277    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="03de250e-9d31-4b1f-b607-1e958954d754" containerName="authorino"
Jul 14 19:03:44.383507 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383287    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9106aa0f-555d-491d-8f35-c38f81e4c040" containerName="authorino"
Jul 14 19:03:44.383507 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.383296    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73ace37-b86b-4a89-9f15-3c53f19b5d4b" containerName="authorino"
Jul 14 19:03:44.385222 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.385204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.387951 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.387929    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 14 19:03:44.388892 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.388874    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fx8vd\""
Jul 14 19:03:44.388987 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.388905    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 19:03:44.388987 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.388901    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 19:03:44.402054 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.402026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"]
Jul 14 19:03:44.419196 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.419167    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:44.527830 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.527788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.528021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.527861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbhb\" (UniqueName: \"kubernetes.io/projected/34778d9e-c272-4861-8884-d06f291d0742-kube-api-access-knbhb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.528021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.527899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34778d9e-c272-4861-8884-d06f291d0742-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.528021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.527928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.528021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.527975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.528210 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.528039    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629265 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbhb\" (UniqueName: \"kubernetes.io/projected/34778d9e-c272-4861-8884-d06f291d0742-kube-api-access-knbhb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34778d9e-c272-4861-8884-d06f291d0742-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629684 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629758 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.629824 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.629803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.631605 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.631579    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34778d9e-c272-4861-8884-d06f291d0742-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.631840 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.631825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34778d9e-c272-4861-8884-d06f291d0742-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.638895 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.638843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbhb\" (UniqueName: \"kubernetes.io/projected/34778d9e-c272-4861-8884-d06f291d0742-kube-api-access-knbhb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cctr8\" (UID: \"34778d9e-c272-4861-8884-d06f291d0742\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.695918 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.695876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:44.837071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:44.837048    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"]
Jul 14 19:03:44.839270 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:03:44.839231    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34778d9e_c272_4861_8884_d06f291d0742.slice/crio-c6d589e7b4baafc9521ca40a49bafd1b1c011abc8490491279c047b61f3d511c WatchSource:0}: Error finding container c6d589e7b4baafc9521ca40a49bafd1b1c011abc8490491279c047b61f3d511c: Status 404 returned error can't find the container with id c6d589e7b4baafc9521ca40a49bafd1b1c011abc8490491279c047b61f3d511c
Jul 14 19:03:45.041267 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:45.041228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8" event={"ID":"34778d9e-c272-4861-8884-d06f291d0742","Type":"ContainerStarted","Data":"c6d589e7b4baafc9521ca40a49bafd1b1c011abc8490491279c047b61f3d511c"}
Jul 14 19:03:47.053637 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:47.053603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8" event={"ID":"34778d9e-c272-4861-8884-d06f291d0742","Type":"ContainerStarted","Data":"31aa62c2f33559293b67a979c518afd5e6010ce1ab6bc380984cb092629606e8"}
Jul 14 19:03:47.077074 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:47.077020    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8" podStartSLOduration=1.226452558 podStartE2EDuration="3.077002554s" totalImagesPullingTime="1.850549996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:03:47.075074361 +0000 UTC m=+570.947632581" watchObservedRunningTime="2026-07-14 19:03:47.077002554 +0000 UTC m=+570.949560774"
Jul 14 19:03:49.472014 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.471982    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"]
Jul 14 19:03:49.474518 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.474493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.477153 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.477136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 14 19:03:49.491599 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.491575    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"]
Jul 14 19:03:49.549558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.549527    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:49.574701 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.574652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.574701 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.574704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55zw\" (UniqueName: \"kubernetes.io/projected/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-kube-api-access-r55zw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.574930 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.574784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.574930 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.574806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.574930 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.574842    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.574930 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.574855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676308 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676308 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676565 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676565 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676565 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676565 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55zw\" (UniqueName: \"kubernetes.io/projected/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-kube-api-access-r55zw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676774 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676865 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676824    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.676865 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.676836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.678905 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.678883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.679048 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.679030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.686295 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.686274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55zw\" (UniqueName: \"kubernetes.io/projected/86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4-kube-api-access-r55zw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bdfff\" (UID: \"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.784642 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.784550    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:49.921932 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:49.921893    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"]
Jul 14 19:03:49.923774 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:03:49.923743    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a50ecc_56e8_4ee1_b654_a6c4a7b5bbd4.slice/crio-e7008b6a1e22270b82c205e669681f363243d58822c7a337bc0cbd41f11ee45f WatchSource:0}: Error finding container e7008b6a1e22270b82c205e669681f363243d58822c7a337bc0cbd41f11ee45f: Status 404 returned error can't find the container with id e7008b6a1e22270b82c205e669681f363243d58822c7a337bc0cbd41f11ee45f
Jul 14 19:03:50.067208 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:50.067122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff" event={"ID":"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4","Type":"ContainerStarted","Data":"e7008b6a1e22270b82c205e669681f363243d58822c7a337bc0cbd41f11ee45f"}
Jul 14 19:03:50.113421 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:50.113364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:51.073464 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:51.073421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff" event={"ID":"86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4","Type":"ContainerStarted","Data":"d6f83624eef6dea99a9d21342c0b03dec5564f27752ed08e76ba56ae01fdc631"}
Jul 14 19:03:51.096550 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:51.096497    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff" podStartSLOduration=1.877751481 podStartE2EDuration="2.096481395s" totalImagesPullingTime="218.729914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:03:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:03:51.09410763 +0000 UTC m=+574.966665851" watchObservedRunningTime="2026-07-14 19:03:51.096481395 +0000 UTC m=+574.969039628"
Jul 14 19:03:52.962363 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:52.962318    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:54.697004 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:54.696967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:54.697446 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:54.697052    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:54.712084 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:54.712055    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:55.103162 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:55.103113    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cctr8"
Jul 14 19:03:55.128877 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:55.128838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:03:59.785324 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:59.785286    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:59.785788 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:59.785345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:03:59.798059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:03:59.798030    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:04:00.127266 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:00.127190    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bdfff"
Jul 14 19:04:03.875993 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:03.875955    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:04:08.576606 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:08.576572    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:04:19.478351 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.478317    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"]
Jul 14 19:04:19.480777 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.480758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.485108 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.485088    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 19:04:19.494211 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.494185    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"]
Jul 14 19:04:19.538916 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.538882    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:04:19.545921 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.545894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.546087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.545930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47be0a3e-e410-4219-bd1f-9061d126eacb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.546087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.545949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.546087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.545967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.546087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.545984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbldm\" (UniqueName: \"kubernetes.io/projected/47be0a3e-e410-4219-bd1f-9061d126eacb-kube-api-access-pbldm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.546257 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.546109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647211 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47be0a3e-e410-4219-bd1f-9061d126eacb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647211 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647211 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647319 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbldm\" (UniqueName: \"kubernetes.io/projected/47be0a3e-e410-4219-bd1f-9061d126eacb-kube-api-access-pbldm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647494 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647573 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.647573 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.647545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.649513 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.649491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47be0a3e-e410-4219-bd1f-9061d126eacb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.649680 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.649661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47be0a3e-e410-4219-bd1f-9061d126eacb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.656418 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.656377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbldm\" (UniqueName: \"kubernetes.io/projected/47be0a3e-e410-4219-bd1f-9061d126eacb-kube-api-access-pbldm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw\" (UID: \"47be0a3e-e410-4219-bd1f-9061d126eacb\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.790579 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.790484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:19.931719 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:19.931691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"]
Jul 14 19:04:19.935754 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:04:19.935718    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47be0a3e_e410_4219_bd1f_9061d126eacb.slice/crio-d843082c0936511a58c0a293798cb2e2c35e7a5e30620ab0c77eaf2b7fce984a WatchSource:0}: Error finding container d843082c0936511a58c0a293798cb2e2c35e7a5e30620ab0c77eaf2b7fce984a: Status 404 returned error can't find the container with id d843082c0936511a58c0a293798cb2e2c35e7a5e30620ab0c77eaf2b7fce984a
Jul 14 19:04:20.202491 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:20.202373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw" event={"ID":"47be0a3e-e410-4219-bd1f-9061d126eacb","Type":"ContainerStarted","Data":"d843082c0936511a58c0a293798cb2e2c35e7a5e30620ab0c77eaf2b7fce984a"}
Jul 14 19:04:22.212876 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:22.212830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw" event={"ID":"47be0a3e-e410-4219-bd1f-9061d126eacb","Type":"ContainerStarted","Data":"e322748c155c7db156ab42c5e99290e208249dc964b6c2d2dfe9b3a939fc244d"}
Jul 14 19:04:22.234580 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:22.234530    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw" podStartSLOduration=1.81497363 podStartE2EDuration="3.234514078s" totalImagesPullingTime="1.419540448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:04:22.232977451 +0000 UTC m=+606.105535673" watchObservedRunningTime="2026-07-14 19:04:22.234514078 +0000 UTC m=+606.107072299"
Jul 14 19:04:29.790964 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:29.790927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:29.791373 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:29.790979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:29.803502 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:29.803479    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:30.259934 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:30.259911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw"
Jul 14 19:04:48.515893 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:48.515816    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:04:48.649672 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:48.649635    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:04:48.999288 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:48.999255    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:04:49.142311 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:04:49.142272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:05:29.283032 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.282996    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66bc479cdc-758wm"]
Jul 14 19:05:29.286553 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.286534    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.299694 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.299666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66bc479cdc-758wm"]
Jul 14 19:05:29.397996 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.397955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpclc\" (UniqueName: \"kubernetes.io/projected/14f4db86-596c-4d18-83a7-7ca3ece00d77-kube-api-access-hpclc\") pod \"authorino-66bc479cdc-758wm\" (UID: \"14f4db86-596c-4d18-83a7-7ca3ece00d77\") " pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.397996 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.397999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14f4db86-596c-4d18-83a7-7ca3ece00d77-tls-cert\") pod \"authorino-66bc479cdc-758wm\" (UID: \"14f4db86-596c-4d18-83a7-7ca3ece00d77\") " pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.498479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.498429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpclc\" (UniqueName: \"kubernetes.io/projected/14f4db86-596c-4d18-83a7-7ca3ece00d77-kube-api-access-hpclc\") pod \"authorino-66bc479cdc-758wm\" (UID: \"14f4db86-596c-4d18-83a7-7ca3ece00d77\") " pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.498647 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.498489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14f4db86-596c-4d18-83a7-7ca3ece00d77-tls-cert\") pod \"authorino-66bc479cdc-758wm\" (UID: \"14f4db86-596c-4d18-83a7-7ca3ece00d77\") " pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.500965 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.500946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14f4db86-596c-4d18-83a7-7ca3ece00d77-tls-cert\") pod \"authorino-66bc479cdc-758wm\" (UID: \"14f4db86-596c-4d18-83a7-7ca3ece00d77\") " pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.516775 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.516741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpclc\" (UniqueName: \"kubernetes.io/projected/14f4db86-596c-4d18-83a7-7ca3ece00d77-kube-api-access-hpclc\") pod \"authorino-66bc479cdc-758wm\" (UID: \"14f4db86-596c-4d18-83a7-7ca3ece00d77\") " pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.602586 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.602500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66bc479cdc-758wm"
Jul 14 19:05:29.763043 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.763013    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66bc479cdc-758wm"]
Jul 14 19:05:29.764738 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:05:29.764653    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14f4db86_596c_4d18_83a7_7ca3ece00d77.slice/crio-cf237bde890968fc011515f8407e000fa3acce083144850b578184aa51fb115e WatchSource:0}: Error finding container cf237bde890968fc011515f8407e000fa3acce083144850b578184aa51fb115e: Status 404 returned error can't find the container with id cf237bde890968fc011515f8407e000fa3acce083144850b578184aa51fb115e
Jul 14 19:05:29.766673 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.766651    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:05:29.929439 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:29.929322    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:05:30.533764 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:30.533663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bc479cdc-758wm" event={"ID":"14f4db86-596c-4d18-83a7-7ca3ece00d77","Type":"ContainerStarted","Data":"4aa9b5a026221c5a13c87893b0b1a461ca46fed6c244f7e5b234b6688e163fc8"}
Jul 14 19:05:30.533764 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:30.533716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66bc479cdc-758wm" event={"ID":"14f4db86-596c-4d18-83a7-7ca3ece00d77","Type":"ContainerStarted","Data":"cf237bde890968fc011515f8407e000fa3acce083144850b578184aa51fb115e"}
Jul 14 19:05:30.562439 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:30.562367    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66bc479cdc-758wm" podStartSLOduration=1.062855165 podStartE2EDuration="1.56234918s" totalImagesPullingTime="499.494015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:05:30.560185312 +0000 UTC m=+674.432743533" watchObservedRunningTime="2026-07-14 19:05:30.56234918 +0000 UTC m=+674.434907455"
Jul 14 19:05:30.604240 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:30.604196    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77f494c9cd-6kc7f"]
Jul 14 19:05:30.605000 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:30.604479    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-77f494c9cd-6kc7f" podUID="93b9b4e5-621c-4771-a2c2-74e732715ea0" containerName="authorino" containerID="cri-o://a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86" gracePeriod=30
Jul 14 19:05:30.879222 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:30.879188    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:05:31.022344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.022312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gncjf\" (UniqueName: \"kubernetes.io/projected/93b9b4e5-621c-4771-a2c2-74e732715ea0-kube-api-access-gncjf\") pod \"93b9b4e5-621c-4771-a2c2-74e732715ea0\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") "
Jul 14 19:05:31.022519 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.022470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93b9b4e5-621c-4771-a2c2-74e732715ea0-tls-cert\") pod \"93b9b4e5-621c-4771-a2c2-74e732715ea0\" (UID: \"93b9b4e5-621c-4771-a2c2-74e732715ea0\") "
Jul 14 19:05:31.024551 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.024520    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b9b4e5-621c-4771-a2c2-74e732715ea0-kube-api-access-gncjf" (OuterVolumeSpecName: "kube-api-access-gncjf") pod "93b9b4e5-621c-4771-a2c2-74e732715ea0" (UID: "93b9b4e5-621c-4771-a2c2-74e732715ea0"). InnerVolumeSpecName "kube-api-access-gncjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:05:31.035869 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.035835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b9b4e5-621c-4771-a2c2-74e732715ea0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "93b9b4e5-621c-4771-a2c2-74e732715ea0" (UID: "93b9b4e5-621c-4771-a2c2-74e732715ea0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:05:31.123865 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.123773    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncjf\" (UniqueName: \"kubernetes.io/projected/93b9b4e5-621c-4771-a2c2-74e732715ea0-kube-api-access-gncjf\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:05:31.123865 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.123814    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93b9b4e5-621c-4771-a2c2-74e732715ea0-tls-cert\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:05:31.540931 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.540899    2562 generic.go:356] "Generic (PLEG): container finished" podID="93b9b4e5-621c-4771-a2c2-74e732715ea0" containerID="a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86" exitCode=0
Jul 14 19:05:31.541344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.540944    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77f494c9cd-6kc7f"
Jul 14 19:05:31.541344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.540991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f494c9cd-6kc7f" event={"ID":"93b9b4e5-621c-4771-a2c2-74e732715ea0","Type":"ContainerDied","Data":"a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86"}
Jul 14 19:05:31.541344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.541038    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77f494c9cd-6kc7f" event={"ID":"93b9b4e5-621c-4771-a2c2-74e732715ea0","Type":"ContainerDied","Data":"a090be2de6ab10d229b1a095b46db602b8ef2603c4d869a89ff9558e43cd0847"}
Jul 14 19:05:31.541344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.541059    2562 scope.go:117] "RemoveContainer" containerID="a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86"
Jul 14 19:05:31.551030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.551008    2562 scope.go:117] "RemoveContainer" containerID="a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86"
Jul 14 19:05:31.551338 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:05:31.551316    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86\": container with ID starting with a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86 not found: ID does not exist" containerID="a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86"
Jul 14 19:05:31.551427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.551347    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86"} err="failed to get container status \"a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86\": rpc error: code = NotFound desc = could not find container \"a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86\": container with ID starting with a8072fa2fc94853b4ac84a922c59f3cd66845013bec5476ce91e1f0dbf848e86 not found: ID does not exist"
Jul 14 19:05:31.567725 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.567695    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77f494c9cd-6kc7f"]
Jul 14 19:05:31.572413 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:31.572375    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77f494c9cd-6kc7f"]
Jul 14 19:05:32.625158 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:32.625106    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b9b4e5-621c-4771-a2c2-74e732715ea0" path="/var/lib/kubelet/pods/93b9b4e5-621c-4771-a2c2-74e732715ea0/volumes"
Jul 14 19:05:34.614483 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:05:34.614446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:06:44.172978 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:06:44.172932    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:06:46.018282 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:06:46.018241    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:30.171740 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:30.171706    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:30.275996 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:30.275965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:30.444363 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:30.444262    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:37.141351 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:37.141302    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:37.557773 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:37.557733    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:37.658808 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:37.658769    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:44.657165 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:44.657135    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:07:45.672922 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:07:45.672890    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:10:28.084395 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:10:28.084356    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:00.205574 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:00.205482    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:06.180912 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:06.180877    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:10.823542 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:10.823491    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:10.884463 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:10.884424    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:10.952794 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:10.952749    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:27.929285 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:27.929245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:47.317177 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:47.317130    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"]
Jul 14 19:11:47.317789 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:47.317424    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6" podUID="ee81bd74-d21d-4ecd-a49d-c3eed90df472" containerName="manager" containerID="cri-o://06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c" gracePeriod=10
Jul 14 19:11:48.281917 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:48.281885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:48.389712 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:48.389679    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:11:50.270549 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.270522    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:11:50.282358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.282330    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee81bd74-d21d-4ecd-a49d-c3eed90df472" containerID="06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c" exitCode=0
Jul 14 19:11:50.282548 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.282371    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"
Jul 14 19:11:50.282548 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.282429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6" event={"ID":"ee81bd74-d21d-4ecd-a49d-c3eed90df472","Type":"ContainerDied","Data":"06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c"}
Jul 14 19:11:50.282548 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.282525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6" event={"ID":"ee81bd74-d21d-4ecd-a49d-c3eed90df472","Type":"ContainerDied","Data":"cdc195b11f9780b35cf9dd2eda8e0f79b4480f903464667d9700b8a75bc30c5f"}
Jul 14 19:11:50.282548 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.282540    2562 scope.go:117] "RemoveContainer" containerID="06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c"
Jul 14 19:11:50.287509 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.287486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee81bd74-d21d-4ecd-a49d-c3eed90df472-extensions-socket-volume\") pod \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") "
Jul 14 19:11:50.287636 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.287524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmgzz\" (UniqueName: \"kubernetes.io/projected/ee81bd74-d21d-4ecd-a49d-c3eed90df472-kube-api-access-rmgzz\") pod \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\" (UID: \"ee81bd74-d21d-4ecd-a49d-c3eed90df472\") "
Jul 14 19:11:50.287857 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.287836    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee81bd74-d21d-4ecd-a49d-c3eed90df472-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee81bd74-d21d-4ecd-a49d-c3eed90df472" (UID: "ee81bd74-d21d-4ecd-a49d-c3eed90df472"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:11:50.290193 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.290152    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee81bd74-d21d-4ecd-a49d-c3eed90df472-kube-api-access-rmgzz" (OuterVolumeSpecName: "kube-api-access-rmgzz") pod "ee81bd74-d21d-4ecd-a49d-c3eed90df472" (UID: "ee81bd74-d21d-4ecd-a49d-c3eed90df472"). InnerVolumeSpecName "kube-api-access-rmgzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:11:50.293649 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.293627    2562 scope.go:117] "RemoveContainer" containerID="06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c"
Jul 14 19:11:50.294089 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:11:50.294062    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c\": container with ID starting with 06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c not found: ID does not exist" containerID="06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c"
Jul 14 19:11:50.294199 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.294101    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c"} err="failed to get container status \"06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c\": rpc error: code = NotFound desc = could not find container \"06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c\": container with ID starting with 06bcaf21efbac037679915ff26f8e9438f31574d362bb7dba92f0cd6e35a271c not found: ID does not exist"
Jul 14 19:11:50.388845 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.388755    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee81bd74-d21d-4ecd-a49d-c3eed90df472-extensions-socket-volume\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:11:50.388845 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.388788    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgzz\" (UniqueName: \"kubernetes.io/projected/ee81bd74-d21d-4ecd-a49d-c3eed90df472-kube-api-access-rmgzz\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:11:50.605638 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.605606    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"]
Jul 14 19:11:50.610785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.610762    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qtw6"]
Jul 14 19:11:50.620603 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:11:50.620577    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee81bd74-d21d-4ecd-a49d-c3eed90df472" path="/var/lib/kubelet/pods/ee81bd74-d21d-4ecd-a49d-c3eed90df472/volumes"
Jul 14 19:13:58.969937 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:13:58.969859    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:13:59.054292 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:13:59.054253    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:15:00.145835 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.145800    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"]
Jul 14 19:15:00.148011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.146133    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee81bd74-d21d-4ecd-a49d-c3eed90df472" containerName="manager"
Jul 14 19:15:00.148011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.146144    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee81bd74-d21d-4ecd-a49d-c3eed90df472" containerName="manager"
Jul 14 19:15:00.148011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.146157    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b9b4e5-621c-4771-a2c2-74e732715ea0" containerName="authorino"
Jul 14 19:15:00.148011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.146163    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b9b4e5-621c-4771-a2c2-74e732715ea0" containerName="authorino"
Jul 14 19:15:00.148011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.146240    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee81bd74-d21d-4ecd-a49d-c3eed90df472" containerName="manager"
Jul 14 19:15:00.148011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.146252    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b9b4e5-621c-4771-a2c2-74e732715ea0" containerName="authorino"
Jul 14 19:15:00.149036 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.149021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"
Jul 14 19:15:00.162441 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.162397    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:15:00.169044 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.169019    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"
Jul 14 19:15:00.183479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.183439    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"]
Jul 14 19:15:00.321735 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.321699    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"]
Jul 14 19:15:00.323822 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:15:00.323794    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010cf5d0_ccea_416a_94d3_05ea84e1e95f.slice/crio-5a1662471d93dbdddef1003c702298c8e9fac050d9673d8b52c685460a52055d WatchSource:0}: Error finding container 5a1662471d93dbdddef1003c702298c8e9fac050d9673d8b52c685460a52055d: Status 404 returned error can't find the container with id 5a1662471d93dbdddef1003c702298c8e9fac050d9673d8b52c685460a52055d
Jul 14 19:15:00.325803 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:00.325786    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:15:01.141007 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:01.140959    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd" event={"ID":"010cf5d0-ccea-416a-94d3-05ea84e1e95f","Type":"ContainerStarted","Data":"5a1662471d93dbdddef1003c702298c8e9fac050d9673d8b52c685460a52055d"}
Jul 14 19:15:03.152955 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:03.152866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd" event={"ID":"010cf5d0-ccea-416a-94d3-05ea84e1e95f","Type":"ContainerStarted","Data":"097cf562292ef6824da489f9ab711bdc620c2a1995f41205d3eacfc401dfb4fb"}
Jul 14 19:15:03.171031 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:03.170984    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd" podStartSLOduration=1.026733589 podStartE2EDuration="3.170967709s" totalImagesPullingTime="2.14423412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:15:03.169626604 +0000 UTC m=+1247.042184829" watchObservedRunningTime="2026-07-14 19:15:03.170967709 +0000 UTC m=+1247.043525943"
Jul 14 19:15:04.158460 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:04.158395    2562 generic.go:356] "Generic (PLEG): container finished" podID="010cf5d0-ccea-416a-94d3-05ea84e1e95f" containerID="097cf562292ef6824da489f9ab711bdc620c2a1995f41205d3eacfc401dfb4fb" exitCode=0
Jul 14 19:15:04.158460 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:04.158440    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd" event={"ID":"010cf5d0-ccea-416a-94d3-05ea84e1e95f","Type":"ContainerDied","Data":"097cf562292ef6824da489f9ab711bdc620c2a1995f41205d3eacfc401dfb4fb"}
Jul 14 19:15:05.291059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:05.291038    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"
Jul 14 19:15:06.168270 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:06.168246    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd"
Jul 14 19:15:06.168465 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:06.168243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734275-srpzd" event={"ID":"010cf5d0-ccea-416a-94d3-05ea84e1e95f","Type":"ContainerDied","Data":"5a1662471d93dbdddef1003c702298c8e9fac050d9673d8b52c685460a52055d"}
Jul 14 19:15:06.168465 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:15:06.168371    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a1662471d93dbdddef1003c702298c8e9fac050d9673d8b52c685460a52055d"
Jul 14 19:18:42.022868 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:18:42.022833    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:18:49.437669 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:18:49.437634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:18:57.702971 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:18:57.702921    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:02.795545 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:02.795509    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:05.727175 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:05.727137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:05.826222 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:05.826189    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:06.156421 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:06.156304    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:08.399526 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:08.399486    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:12.297673 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:12.297611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:12.398252 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:12.398210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:12.522884 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:12.522838    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:12.623151 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:12.623067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:13.829577 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.829542    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"]
Jul 14 19:19:13.830118 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.830095    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010cf5d0-ccea-416a-94d3-05ea84e1e95f" containerName="cleanup"
Jul 14 19:19:13.830178 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.830122    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="010cf5d0-ccea-416a-94d3-05ea84e1e95f" containerName="cleanup"
Jul 14 19:19:13.830245 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.830230    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="010cf5d0-ccea-416a-94d3-05ea84e1e95f" containerName="cleanup"
Jul 14 19:19:13.832947 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.832922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:13.837905 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.837879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-61c60be8-serving-cert\""
Jul 14 19:19:13.838055 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.837887    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:19:13.838120 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.838110    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:19:13.838206 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.838167    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:19:13.843492 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.843467    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"]
Jul 14 19:19:13.909147 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.909111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxvv\" (UniqueName: \"kubernetes.io/projected/afa501b6-a81b-482a-ace0-8d94837aa2e3-kube-api-access-gtxvv\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:13.909147 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.909156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa501b6-a81b-482a-ace0-8d94837aa2e3-openshift-service-ca\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:13.909435 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:13.909227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.010386 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.010348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxvv\" (UniqueName: \"kubernetes.io/projected/afa501b6-a81b-482a-ace0-8d94837aa2e3-kube-api-access-gtxvv\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.010386 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.010388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa501b6-a81b-482a-ace0-8d94837aa2e3-openshift-service-ca\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.010685 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.010476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.010685 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:19:14.010651    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-serving-cert: secret "maas-api-e2e-ait-61c60be8-serving-cert" not found
Jul 14 19:19:14.010977 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:19:14.010719    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls podName:afa501b6-a81b-482a-ace0-8d94837aa2e3 nodeName:}" failed. No retries permitted until 2026-07-14 19:19:14.510700013 +0000 UTC m=+1498.383258227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls") pod "maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" (UID: "afa501b6-a81b-482a-ace0-8d94837aa2e3") : secret "maas-api-e2e-ait-61c60be8-serving-cert" not found
Jul 14 19:19:14.011230 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.011209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa501b6-a81b-482a-ace0-8d94837aa2e3-openshift-service-ca\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.019355 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.019327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxvv\" (UniqueName: \"kubernetes.io/projected/afa501b6-a81b-482a-ace0-8d94837aa2e3-kube-api-access-gtxvv\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.514765 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.514724    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.517448 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.517428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls\") pod \"maas-api-e2e-ait-61c60be8-6789f6998c-gs57t\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.746358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.746327    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:14.884552 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:14.884526    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"]
Jul 14 19:19:14.886790 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:19:14.886756    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa501b6_a81b_482a_ace0_8d94837aa2e3.slice/crio-66fede4f020456b1fc0d00d4f24dd00611c2c4447595d70693666ed7cf04f6ea WatchSource:0}: Error finding container 66fede4f020456b1fc0d00d4f24dd00611c2c4447595d70693666ed7cf04f6ea: Status 404 returned error can't find the container with id 66fede4f020456b1fc0d00d4f24dd00611c2c4447595d70693666ed7cf04f6ea
Jul 14 19:19:15.299636 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:15.299600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" event={"ID":"afa501b6-a81b-482a-ace0-8d94837aa2e3","Type":"ContainerStarted","Data":"66fede4f020456b1fc0d00d4f24dd00611c2c4447595d70693666ed7cf04f6ea"}
Jul 14 19:19:16.061574 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:16.061523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:16.164091 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:16.164053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:16.340206 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:16.340158    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:19:17.309713 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:17.309675    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" event={"ID":"afa501b6-a81b-482a-ace0-8d94837aa2e3","Type":"ContainerStarted","Data":"0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d"}
Jul 14 19:19:17.310227 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:17.309785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:17.329079 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:17.329032    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" podStartSLOduration=2.347348849 podStartE2EDuration="4.329015982s" totalImagesPullingTime="1.981667133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:19:17.327787355 +0000 UTC m=+1501.200345577" watchObservedRunningTime="2026-07-14 19:19:17.329015982 +0000 UTC m=+1501.201574202"
Jul 14 19:19:18.722223 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.722189    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"]
Jul 14 19:19:18.724738 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.724720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:18.727325 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.727294    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:19:18.734339 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.734313    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"]
Jul 14 19:19:18.853825 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.853790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f41900e-a085-49aa-b5ce-c48683ae78be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n\" (UID: \"0f41900e-a085-49aa-b5ce-c48683ae78be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:18.955190 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.955149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f41900e-a085-49aa-b5ce-c48683ae78be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n\" (UID: \"0f41900e-a085-49aa-b5ce-c48683ae78be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:18.955840 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:18.955818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f41900e-a085-49aa-b5ce-c48683ae78be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n\" (UID: \"0f41900e-a085-49aa-b5ce-c48683ae78be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:19.035781 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:19.035695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:19.169130 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:19.169103    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"]
Jul 14 19:19:19.171316 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:19:19.171281    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f41900e_a085_49aa_b5ce_c48683ae78be.slice/crio-5ceaede7851919ed1ba6edc23296dc6d4b3cd23eb278110e777c0aeffc43febd WatchSource:0}: Error finding container 5ceaede7851919ed1ba6edc23296dc6d4b3cd23eb278110e777c0aeffc43febd: Status 404 returned error can't find the container with id 5ceaede7851919ed1ba6edc23296dc6d4b3cd23eb278110e777c0aeffc43febd
Jul 14 19:19:19.322254 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:19.322173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" event={"ID":"0f41900e-a085-49aa-b5ce-c48683ae78be","Type":"ContainerStarted","Data":"5ceaede7851919ed1ba6edc23296dc6d4b3cd23eb278110e777c0aeffc43febd"}
Jul 14 19:19:20.328029 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:20.327986    2562 generic.go:356] "Generic (PLEG): container finished" podID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerID="2a166ce375023f2dcdc4f60b1b9da9415d315e7ece3c56ab4bf4a6331a05a18a" exitCode=7
Jul 14 19:19:20.328435 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:20.328110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" event={"ID":"0f41900e-a085-49aa-b5ce-c48683ae78be","Type":"ContainerDied","Data":"2a166ce375023f2dcdc4f60b1b9da9415d315e7ece3c56ab4bf4a6331a05a18a"}
Jul 14 19:19:20.328435 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:20.328237    2562 scope.go:117] "RemoveContainer" containerID="2a166ce375023f2dcdc4f60b1b9da9415d315e7ece3c56ab4bf4a6331a05a18a"
Jul 14 19:19:21.335148 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:21.335112    2562 generic.go:356] "Generic (PLEG): container finished" podID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerID="0cc4619ae69f029101b35f4aced7eb5ad55f879c87805adcd0386ac6443dc16a" exitCode=7
Jul 14 19:19:21.335674 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:21.335163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" event={"ID":"0f41900e-a085-49aa-b5ce-c48683ae78be","Type":"ContainerDied","Data":"0cc4619ae69f029101b35f4aced7eb5ad55f879c87805adcd0386ac6443dc16a"}
Jul 14 19:19:21.335674 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:21.335224    2562 scope.go:117] "RemoveContainer" containerID="2a166ce375023f2dcdc4f60b1b9da9415d315e7ece3c56ab4bf4a6331a05a18a"
Jul 14 19:19:21.335674 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:21.335374    2562 scope.go:117] "RemoveContainer" containerID="0cc4619ae69f029101b35f4aced7eb5ad55f879c87805adcd0386ac6443dc16a"
Jul 14 19:19:21.335674 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:19:21.335650    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n_odh-ai-gateway-infra(0f41900e-a085-49aa-b5ce-c48683ae78be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be"
Jul 14 19:19:22.341092 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:22.341059    2562 scope.go:117] "RemoveContainer" containerID="0cc4619ae69f029101b35f4aced7eb5ad55f879c87805adcd0386ac6443dc16a"
Jul 14 19:19:22.341576 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:19:22.341313    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n_odh-ai-gateway-infra(0f41900e-a085-49aa-b5ce-c48683ae78be)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be"
Jul 14 19:19:23.323322 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:23.323295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:19:37.615777 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:37.615698    2562 scope.go:117] "RemoveContainer" containerID="0cc4619ae69f029101b35f4aced7eb5ad55f879c87805adcd0386ac6443dc16a"
Jul 14 19:19:38.418846 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:38.418801    2562 generic.go:356] "Generic (PLEG): container finished" podID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerID="6e422d3f056ff5707febb031ed7dd408757495eac0829c6bc0c8cbebfc54b8c5" exitCode=0
Jul 14 19:19:38.419011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:38.418873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" event={"ID":"0f41900e-a085-49aa-b5ce-c48683ae78be","Type":"ContainerDied","Data":"6e422d3f056ff5707febb031ed7dd408757495eac0829c6bc0c8cbebfc54b8c5"}
Jul 14 19:19:38.419011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:38.418966    2562 scope.go:117] "RemoveContainer" containerID="0cc4619ae69f029101b35f4aced7eb5ad55f879c87805adcd0386ac6443dc16a"
Jul 14 19:19:39.452243 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:39.452212    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"]
Jul 14 19:19:39.552416 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:39.552381    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:39.640575 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:39.640538    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f41900e-a085-49aa-b5ce-c48683ae78be-maas-api-service-ca\") pod \"0f41900e-a085-49aa-b5ce-c48683ae78be\" (UID: \"0f41900e-a085-49aa-b5ce-c48683ae78be\") "
Jul 14 19:19:39.640888 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:39.640867    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f41900e-a085-49aa-b5ce-c48683ae78be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f41900e-a085-49aa-b5ce-c48683ae78be" (UID: "0f41900e-a085-49aa-b5ce-c48683ae78be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:19:39.741652 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:39.741617    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f41900e-a085-49aa-b5ce-c48683ae78be-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:19:40.430316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:40.430287    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"
Jul 14 19:19:40.430316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:40.430300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n" event={"ID":"0f41900e-a085-49aa-b5ce-c48683ae78be","Type":"ContainerDied","Data":"5ceaede7851919ed1ba6edc23296dc6d4b3cd23eb278110e777c0aeffc43febd"}
Jul 14 19:19:40.430543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:40.430347    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ceaede7851919ed1ba6edc23296dc6d4b3cd23eb278110e777c0aeffc43febd"
Jul 14 19:19:40.459585 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:40.459552    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"]
Jul 14 19:19:40.465472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:40.465441    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9jb4n"]
Jul 14 19:19:40.621065 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:40.621027    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" path="/var/lib/kubelet/pods/0f41900e-a085-49aa-b5ce-c48683ae78be/volumes"
Jul 14 19:19:50.932497 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:19:50.932463    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:01.690591 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:01.690549    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:11.566807 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:11.566763    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:18.777251 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777209    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"]
Jul 14 19:20:18.777656 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777565    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.777656 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777576    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.777656 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777586    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.777656 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777592    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.777801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777660    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.777801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777667    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.777801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.777673    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:18.780584 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.780568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:18.783596 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.783571    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:20:18.795726 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.795705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"]
Jul 14 19:20:18.806337 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.806311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7081b84-acd7-476e-8a0e-10582624f2b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp\" (UID: \"f7081b84-acd7-476e-8a0e-10582624f2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:18.906970 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.906937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7081b84-acd7-476e-8a0e-10582624f2b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp\" (UID: \"f7081b84-acd7-476e-8a0e-10582624f2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:18.907604 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:18.907578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7081b84-acd7-476e-8a0e-10582624f2b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp\" (UID: \"f7081b84-acd7-476e-8a0e-10582624f2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:19.090043 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:19.089960    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:19.225966 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:19.225939    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"]
Jul 14 19:20:19.227736 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:20:19.227706    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7081b84_acd7_476e_8a0e_10582624f2b4.slice/crio-7bbed2fe143a1f0a96db051ef9f23286d7a69259b23e81199299a882fd164740 WatchSource:0}: Error finding container 7bbed2fe143a1f0a96db051ef9f23286d7a69259b23e81199299a882fd164740: Status 404 returned error can't find the container with id 7bbed2fe143a1f0a96db051ef9f23286d7a69259b23e81199299a882fd164740
Jul 14 19:20:19.612424 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:19.612363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp" event={"ID":"f7081b84-acd7-476e-8a0e-10582624f2b4","Type":"ContainerStarted","Data":"7693dacf9b7cb1c4bc80e9ccf4606f0d1fa6e5c5aa4d4b71189229f48444201d"}
Jul 14 19:20:19.612615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:19.612441    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp" event={"ID":"f7081b84-acd7-476e-8a0e-10582624f2b4","Type":"ContainerStarted","Data":"7bbed2fe143a1f0a96db051ef9f23286d7a69259b23e81199299a882fd164740"}
Jul 14 19:20:19.631300 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:19.631254    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp" podStartSLOduration=1.631237965 podStartE2EDuration="1.631237965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:20:19.629598137 +0000 UTC m=+1563.502156357" watchObservedRunningTime="2026-07-14 19:20:19.631237965 +0000 UTC m=+1563.503796185"
Jul 14 19:20:20.618777 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:20.618731    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7081b84-acd7-476e-8a0e-10582624f2b4" containerID="7693dacf9b7cb1c4bc80e9ccf4606f0d1fa6e5c5aa4d4b71189229f48444201d" exitCode=0
Jul 14 19:20:20.620781 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:20.620755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp" event={"ID":"f7081b84-acd7-476e-8a0e-10582624f2b4","Type":"ContainerDied","Data":"7693dacf9b7cb1c4bc80e9ccf4606f0d1fa6e5c5aa4d4b71189229f48444201d"}
Jul 14 19:20:21.142909 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:21.142869    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:21.243729 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:21.243691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:21.755745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:21.755720    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:21.833072 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:21.833039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7081b84-acd7-476e-8a0e-10582624f2b4-maas-api-service-ca\") pod \"f7081b84-acd7-476e-8a0e-10582624f2b4\" (UID: \"f7081b84-acd7-476e-8a0e-10582624f2b4\") "
Jul 14 19:20:21.833397 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:21.833374    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7081b84-acd7-476e-8a0e-10582624f2b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7081b84-acd7-476e-8a0e-10582624f2b4" (UID: "f7081b84-acd7-476e-8a0e-10582624f2b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:20:21.933973 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:21.933878    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7081b84-acd7-476e-8a0e-10582624f2b4-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:20:22.628315 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:22.628280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp" event={"ID":"f7081b84-acd7-476e-8a0e-10582624f2b4","Type":"ContainerDied","Data":"7bbed2fe143a1f0a96db051ef9f23286d7a69259b23e81199299a882fd164740"}
Jul 14 19:20:22.628476 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:22.628325    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bbed2fe143a1f0a96db051ef9f23286d7a69259b23e81199299a882fd164740"
Jul 14 19:20:22.628476 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:22.628285    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp"
Jul 14 19:20:29.263378 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.263340    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"]
Jul 14 19:20:29.263886 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.263607    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" podUID="afa501b6-a81b-482a-ace0-8d94837aa2e3" containerName="maas-api" containerID="cri-o://0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d" gracePeriod=30
Jul 14 19:20:29.297044 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.297018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:29.518859 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.518800    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:20:29.605209 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.605167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa501b6-a81b-482a-ace0-8d94837aa2e3-openshift-service-ca\") pod \"afa501b6-a81b-482a-ace0-8d94837aa2e3\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") "
Jul 14 19:20:29.605209 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.605202    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls\") pod \"afa501b6-a81b-482a-ace0-8d94837aa2e3\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") "
Jul 14 19:20:29.605510 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.605257    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtxvv\" (UniqueName: \"kubernetes.io/projected/afa501b6-a81b-482a-ace0-8d94837aa2e3-kube-api-access-gtxvv\") pod \"afa501b6-a81b-482a-ace0-8d94837aa2e3\" (UID: \"afa501b6-a81b-482a-ace0-8d94837aa2e3\") "
Jul 14 19:20:29.605629 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.605604    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa501b6-a81b-482a-ace0-8d94837aa2e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "afa501b6-a81b-482a-ace0-8d94837aa2e3" (UID: "afa501b6-a81b-482a-ace0-8d94837aa2e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:20:29.607637 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.607604    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "afa501b6-a81b-482a-ace0-8d94837aa2e3" (UID: "afa501b6-a81b-482a-ace0-8d94837aa2e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:20:29.607637 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.607602    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa501b6-a81b-482a-ace0-8d94837aa2e3-kube-api-access-gtxvv" (OuterVolumeSpecName: "kube-api-access-gtxvv") pod "afa501b6-a81b-482a-ace0-8d94837aa2e3" (UID: "afa501b6-a81b-482a-ace0-8d94837aa2e3"). InnerVolumeSpecName "kube-api-access-gtxvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:20:29.662934 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.662898    2562 generic.go:356] "Generic (PLEG): container finished" podID="afa501b6-a81b-482a-ace0-8d94837aa2e3" containerID="0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d" exitCode=0
Jul 14 19:20:29.663095 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.662945    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"
Jul 14 19:20:29.663095 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.662992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" event={"ID":"afa501b6-a81b-482a-ace0-8d94837aa2e3","Type":"ContainerDied","Data":"0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d"}
Jul 14 19:20:29.663095 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.663050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t" event={"ID":"afa501b6-a81b-482a-ace0-8d94837aa2e3","Type":"ContainerDied","Data":"66fede4f020456b1fc0d00d4f24dd00611c2c4447595d70693666ed7cf04f6ea"}
Jul 14 19:20:29.663095 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.663078    2562 scope.go:117] "RemoveContainer" containerID="0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d"
Jul 14 19:20:29.671560 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.671544    2562 scope.go:117] "RemoveContainer" containerID="0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d"
Jul 14 19:20:29.671816 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:20:29.671798    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d\": container with ID starting with 0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d not found: ID does not exist" containerID="0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d"
Jul 14 19:20:29.671863 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.671827    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d"} err="failed to get container status \"0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d\": rpc error: code = NotFound desc = could not find container \"0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d\": container with ID starting with 0735164c45d41132345511f09390c2136c7ffdf381b50acb5f28a819f8960b8d not found: ID does not exist"
Jul 14 19:20:29.685977 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.685950    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"]
Jul 14 19:20:29.690021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.689996    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61c60be8-6789f6998c-gs57t"]
Jul 14 19:20:29.706778 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.706751    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxvv\" (UniqueName: \"kubernetes.io/projected/afa501b6-a81b-482a-ace0-8d94837aa2e3-kube-api-access-gtxvv\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:20:29.706888 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.706784    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa501b6-a81b-482a-ace0-8d94837aa2e3-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:20:29.706888 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:29.706800    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa501b6-a81b-482a-ace0-8d94837aa2e3-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:20:30.622241 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:30.622206    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afa501b6-a81b-482a-ace0-8d94837aa2e3" path="/var/lib/kubelet/pods/afa501b6-a81b-482a-ace0-8d94837aa2e3/volumes"
Jul 14 19:20:33.937950 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:33.937870    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:39.869168 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:39.869137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:39.934588 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:39.934536    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:44.094591 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:44.094554    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:46.044627 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044591    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"]
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044906    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7081b84-acd7-476e-8a0e-10582624f2b4" containerName="revoke-keys"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044916    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7081b84-acd7-476e-8a0e-10582624f2b4" containerName="revoke-keys"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044933    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa501b6-a81b-482a-ace0-8d94837aa2e3" containerName="maas-api"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044939    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa501b6-a81b-482a-ace0-8d94837aa2e3" containerName="maas-api"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044947    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.044953    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f41900e-a085-49aa-b5ce-c48683ae78be" containerName="revoke-keys"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.045003    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7081b84-acd7-476e-8a0e-10582624f2b4" containerName="revoke-keys"
Jul 14 19:20:46.045034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.045012    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="afa501b6-a81b-482a-ace0-8d94837aa2e3" containerName="maas-api"
Jul 14 19:20:46.047838 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.047819    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.055671 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.055642    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:20:46.055992 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.055730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-040d121d-serving-cert\""
Jul 14 19:20:46.056169 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.055957    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:20:46.056777 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.056757    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:20:46.058943 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.058920    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"]
Jul 14 19:20:46.157994 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.157958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-openshift-service-ca\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.158158 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.158013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.158158 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.158111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bks5r\" (UniqueName: \"kubernetes.io/projected/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-kube-api-access-bks5r\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.259452 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.259416    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-openshift-service-ca\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.259624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.259472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.259624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.259516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bks5r\" (UniqueName: \"kubernetes.io/projected/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-kube-api-access-bks5r\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.259709 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:20:46.259647    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-serving-cert: secret "maas-api-e2e-migrate-040d121d-serving-cert" not found
Jul 14 19:20:46.259751 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:20:46.259724    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls podName:9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb nodeName:}" failed. No retries permitted until 2026-07-14 19:20:46.759706525 +0000 UTC m=+1590.632264724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls") pod "maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" (UID: "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb") : secret "maas-api-e2e-migrate-040d121d-serving-cert" not found
Jul 14 19:20:46.260031 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.260010    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-openshift-service-ca\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.268522 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.268498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bks5r\" (UniqueName: \"kubernetes.io/projected/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-kube-api-access-bks5r\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.765168 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.765128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.767801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.767776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls\") pod \"maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:46.840952 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.840909    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:46.966422 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:46.966376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:47.114709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:47.114673    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"]
Jul 14 19:20:47.117743 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:20:47.117699    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbb0486_05f1_4158_bb42_bb7d1f7ff4eb.slice/crio-c78913fe8349fe3d2b8fe144f0fd32983d033a51beeb008f483b76ed31c7647c WatchSource:0}: Error finding container c78913fe8349fe3d2b8fe144f0fd32983d033a51beeb008f483b76ed31c7647c: Status 404 returned error can't find the container with id c78913fe8349fe3d2b8fe144f0fd32983d033a51beeb008f483b76ed31c7647c
Jul 14 19:20:47.119205 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:47.119184    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:20:47.752983 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:47.752944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" event={"ID":"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb","Type":"ContainerStarted","Data":"c78913fe8349fe3d2b8fe144f0fd32983d033a51beeb008f483b76ed31c7647c"}
Jul 14 19:20:48.758502 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:48.758471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" event={"ID":"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb","Type":"ContainerStarted","Data":"b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a"}
Jul 14 19:20:48.758877 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:48.758580    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:48.781885 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:48.781837    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" podStartSLOduration=2.097456575 podStartE2EDuration="2.781821619s" totalImagesPullingTime="684.365044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:20:48.78110188 +0000 UTC m=+1592.653660101" watchObservedRunningTime="2026-07-14 19:20:48.781821619 +0000 UTC m=+1592.654379838"
Jul 14 19:20:49.168876 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:49.168779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:50.378062 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.378022    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"]
Jul 14 19:20:50.381697 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.381674    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:20:50.384396 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.384370    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:20:50.398035 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.398005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"]
Jul 14 19:20:50.494774 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.494739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cbd601-6600-40e6-9d57-07afb9d1bdc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-040d121d-vm29f\" (UID: \"27cbd601-6600-40e6-9d57-07afb9d1bdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:20:50.595882 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.595846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cbd601-6600-40e6-9d57-07afb9d1bdc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-040d121d-vm29f\" (UID: \"27cbd601-6600-40e6-9d57-07afb9d1bdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:20:50.596469 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.596451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cbd601-6600-40e6-9d57-07afb9d1bdc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-040d121d-vm29f\" (UID: \"27cbd601-6600-40e6-9d57-07afb9d1bdc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:20:50.691833 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.691736    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:20:50.838106 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:50.830927    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"]
Jul 14 19:20:51.775628 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:51.775589    2562 generic.go:356] "Generic (PLEG): container finished" podID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerID="cddc955c0339a1c8ddeb19557abef071255d193b36141823d17fdc26d51b9560" exitCode=7
Jul 14 19:20:51.776128 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:51.775676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" event={"ID":"27cbd601-6600-40e6-9d57-07afb9d1bdc6","Type":"ContainerDied","Data":"cddc955c0339a1c8ddeb19557abef071255d193b36141823d17fdc26d51b9560"}
Jul 14 19:20:51.776128 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:51.775723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" event={"ID":"27cbd601-6600-40e6-9d57-07afb9d1bdc6","Type":"ContainerStarted","Data":"4e698530ccd4877a4acf462c8406737b529f5c403b88a144de8f23d62df3dd64"}
Jul 14 19:20:51.776128 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:51.775874    2562 scope.go:117] "RemoveContainer" containerID="cddc955c0339a1c8ddeb19557abef071255d193b36141823d17fdc26d51b9560"
Jul 14 19:20:52.781965 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:52.781929    2562 generic.go:356] "Generic (PLEG): container finished" podID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerID="9ddb8323f1daa44afd57422d0ddd152ab5679be5b2076c42fa8ae9b2bda96ae7" exitCode=7
Jul 14 19:20:52.782472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:52.781992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" event={"ID":"27cbd601-6600-40e6-9d57-07afb9d1bdc6","Type":"ContainerDied","Data":"9ddb8323f1daa44afd57422d0ddd152ab5679be5b2076c42fa8ae9b2bda96ae7"}
Jul 14 19:20:52.782472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:52.782037    2562 scope.go:117] "RemoveContainer" containerID="cddc955c0339a1c8ddeb19557abef071255d193b36141823d17fdc26d51b9560"
Jul 14 19:20:52.782472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:52.782185    2562 scope.go:117] "RemoveContainer" containerID="9ddb8323f1daa44afd57422d0ddd152ab5679be5b2076c42fa8ae9b2bda96ae7"
Jul 14 19:20:52.782472 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:20:52.782425    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-040d121d-vm29f_odh-ai-gateway-infra(27cbd601-6600-40e6-9d57-07afb9d1bdc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6"
Jul 14 19:20:53.788813 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:53.788790    2562 scope.go:117] "RemoveContainer" containerID="9ddb8323f1daa44afd57422d0ddd152ab5679be5b2076c42fa8ae9b2bda96ae7"
Jul 14 19:20:53.789173 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:20:53.789015    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-040d121d-vm29f_odh-ai-gateway-infra(27cbd601-6600-40e6-9d57-07afb9d1bdc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6"
Jul 14 19:20:54.769093 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:54.769064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:20:56.139325 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:56.139286    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:56.201852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:56.201810    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:20:59.865263 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:20:59.865206    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:21:09.615785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:09.615705    2562 scope.go:117] "RemoveContainer" containerID="9ddb8323f1daa44afd57422d0ddd152ab5679be5b2076c42fa8ae9b2bda96ae7"
Jul 14 19:21:10.873534 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:10.873498    2562 generic.go:356] "Generic (PLEG): container finished" podID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerID="8bc6ede383a62ffaf60eac6338f1d7e632cb316a1e8813e802a94603f5464a34" exitCode=0
Jul 14 19:21:10.874012 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:10.873568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" event={"ID":"27cbd601-6600-40e6-9d57-07afb9d1bdc6","Type":"ContainerDied","Data":"8bc6ede383a62ffaf60eac6338f1d7e632cb316a1e8813e802a94603f5464a34"}
Jul 14 19:21:10.874012 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:10.873627    2562 scope.go:117] "RemoveContainer" containerID="9ddb8323f1daa44afd57422d0ddd152ab5679be5b2076c42fa8ae9b2bda96ae7"
Jul 14 19:21:11.906828 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:11.906793    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"]
Jul 14 19:21:12.013460 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.013430    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:21:12.097666 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.097629    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cbd601-6600-40e6-9d57-07afb9d1bdc6-maas-api-service-ca\") pod \"27cbd601-6600-40e6-9d57-07afb9d1bdc6\" (UID: \"27cbd601-6600-40e6-9d57-07afb9d1bdc6\") "
Jul 14 19:21:12.098010 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.097986    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cbd601-6600-40e6-9d57-07afb9d1bdc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27cbd601-6600-40e6-9d57-07afb9d1bdc6" (UID: "27cbd601-6600-40e6-9d57-07afb9d1bdc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:21:12.198508 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.198429    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27cbd601-6600-40e6-9d57-07afb9d1bdc6-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:21:12.885920 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.885838    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"
Jul 14 19:21:12.885920 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.885868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f" event={"ID":"27cbd601-6600-40e6-9d57-07afb9d1bdc6","Type":"ContainerDied","Data":"4e698530ccd4877a4acf462c8406737b529f5c403b88a144de8f23d62df3dd64"}
Jul 14 19:21:12.885920 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.885915    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e698530ccd4877a4acf462c8406737b529f5c403b88a144de8f23d62df3dd64"
Jul 14 19:21:12.919285 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.919256    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"]
Jul 14 19:21:12.923985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:12.923958    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-vm29f"]
Jul 14 19:21:14.620767 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:14.620729    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" path="/var/lib/kubelet/pods/27cbd601-6600-40e6-9d57-07afb9d1bdc6/volumes"
Jul 14 19:21:50.430796 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.430761    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"]
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431119    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431130    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431147    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431152    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431213    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431222    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.431262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.431227    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:21:50.434344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.434328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:50.436998 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.436982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:21:50.444121 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.444097    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"]
Jul 14 19:21:50.518459 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.518417    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:21:50.546342 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.546311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1716d-9c75-49f0-a858-137465299a0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-040d121d-wcxns\" (UID: \"10b1716d-9c75-49f0-a858-137465299a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:50.626458 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.626421    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:21:50.647028 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.647000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1716d-9c75-49f0-a858-137465299a0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-040d121d-wcxns\" (UID: \"10b1716d-9c75-49f0-a858-137465299a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:50.647622 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.647604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1716d-9c75-49f0-a858-137465299a0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-040d121d-wcxns\" (UID: \"10b1716d-9c75-49f0-a858-137465299a0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:50.745055 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.745024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:50.912102 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:50.912078    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"]
Jul 14 19:21:50.913928 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:21:50.913900    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b1716d_9c75_49f0_a858_137465299a0a.slice/crio-938888fa4aa19283eaa7ef2df4644960310864f1e807918a8e37c5b9f6645fbe WatchSource:0}: Error finding container 938888fa4aa19283eaa7ef2df4644960310864f1e807918a8e37c5b9f6645fbe: Status 404 returned error can't find the container with id 938888fa4aa19283eaa7ef2df4644960310864f1e807918a8e37c5b9f6645fbe
Jul 14 19:21:51.055105 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:51.055020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns" event={"ID":"10b1716d-9c75-49f0-a858-137465299a0a","Type":"ContainerStarted","Data":"938888fa4aa19283eaa7ef2df4644960310864f1e807918a8e37c5b9f6645fbe"}
Jul 14 19:21:52.060852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:52.060814    2562 generic.go:356] "Generic (PLEG): container finished" podID="10b1716d-9c75-49f0-a858-137465299a0a" containerID="e1b68b11f88f259f231077d63126977779526c6f6010f29b9dc7266b89a8ea2d" exitCode=0
Jul 14 19:21:52.061237 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:52.060903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns" event={"ID":"10b1716d-9c75-49f0-a858-137465299a0a","Type":"ContainerDied","Data":"e1b68b11f88f259f231077d63126977779526c6f6010f29b9dc7266b89a8ea2d"}
Jul 14 19:21:53.195930 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:53.195901    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:53.270506 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:53.270470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1716d-9c75-49f0-a858-137465299a0a-maas-api-service-ca\") pod \"10b1716d-9c75-49f0-a858-137465299a0a\" (UID: \"10b1716d-9c75-49f0-a858-137465299a0a\") "
Jul 14 19:21:53.270844 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:53.270823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b1716d-9c75-49f0-a858-137465299a0a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10b1716d-9c75-49f0-a858-137465299a0a" (UID: "10b1716d-9c75-49f0-a858-137465299a0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:21:53.371454 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:53.371356    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1716d-9c75-49f0-a858-137465299a0a-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:21:54.071700 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:54.071658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns" event={"ID":"10b1716d-9c75-49f0-a858-137465299a0a","Type":"ContainerDied","Data":"938888fa4aa19283eaa7ef2df4644960310864f1e807918a8e37c5b9f6645fbe"}
Jul 14 19:21:54.071700 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:54.071692    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-040d121d-wcxns"
Jul 14 19:21:54.071920 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:54.071717    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="938888fa4aa19283eaa7ef2df4644960310864f1e807918a8e37c5b9f6645fbe"
Jul 14 19:21:56.569443 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.569385    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"]
Jul 14 19:21:56.570007 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.569610    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" podUID="9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" containerName="maas-api" containerID="cri-o://b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a" gracePeriod=30
Jul 14 19:21:56.594919 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.594884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:21:56.812412 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.812373    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:21:56.902325 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.902235    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls\") pod \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") "
Jul 14 19:21:56.902528 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.902348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-openshift-service-ca\") pod \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") "
Jul 14 19:21:56.902528 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.902392    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bks5r\" (UniqueName: \"kubernetes.io/projected/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-kube-api-access-bks5r\") pod \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\" (UID: \"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb\") "
Jul 14 19:21:56.902704 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.902678    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" (UID: "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:21:56.904473 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.904447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" (UID: "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:21:56.904573 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.904504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-kube-api-access-bks5r" (OuterVolumeSpecName: "kube-api-access-bks5r") pod "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" (UID: "9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb"). InnerVolumeSpecName "kube-api-access-bks5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:21:56.943226 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:56.943188    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:21:57.003794 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.003764    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:21:57.003794 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.003791    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:21:57.003995 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.003804    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bks5r\" (UniqueName: \"kubernetes.io/projected/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb-kube-api-access-bks5r\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:21:57.087661 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.087621    2562 generic.go:356] "Generic (PLEG): container finished" podID="9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" containerID="b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a" exitCode=0
Jul 14 19:21:57.087834 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.087673    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"
Jul 14 19:21:57.087834 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.087707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" event={"ID":"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb","Type":"ContainerDied","Data":"b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a"}
Jul 14 19:21:57.087834 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.087760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m" event={"ID":"9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb","Type":"ContainerDied","Data":"c78913fe8349fe3d2b8fe144f0fd32983d033a51beeb008f483b76ed31c7647c"}
Jul 14 19:21:57.087834 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.087782    2562 scope.go:117] "RemoveContainer" containerID="b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a"
Jul 14 19:21:57.096673 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.096656    2562 scope.go:117] "RemoveContainer" containerID="b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a"
Jul 14 19:21:57.096934 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:21:57.096917    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a\": container with ID starting with b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a not found: ID does not exist" containerID="b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a"
Jul 14 19:21:57.096978 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.096945    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a"} err="failed to get container status \"b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a\": rpc error: code = NotFound desc = could not find container \"b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a\": container with ID starting with b4008f01327c96cfcbe315a90daa8fa68ac5438e405f9ff322c1950208c93f7a not found: ID does not exist"
Jul 14 19:21:57.108892 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.108867    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"]
Jul 14 19:21:57.113028 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.112999    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-040d121d-69f4447cb9-rkz7m"]
Jul 14 19:21:57.841270 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:57.841217    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:21:58.622057 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:21:58.622019    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" path="/var/lib/kubelet/pods/9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb/volumes"
Jul 14 19:22:02.667745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:02.667709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:02.767748 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:02.767708    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:02.894007 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:02.893965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:03.002134 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:03.002089    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:04.098261 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.098215    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:04.184476 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184434    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"]
Jul 14 19:22:04.184861 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184845    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:22:04.184861 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184862    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cbd601-6600-40e6-9d57-07afb9d1bdc6" containerName="revoke-keys"
Jul 14 19:22:04.185030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184873    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" containerName="maas-api"
Jul 14 19:22:04.185030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184878    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" containerName="maas-api"
Jul 14 19:22:04.185030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184895    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b1716d-9c75-49f0-a858-137465299a0a" containerName="revoke-keys"
Jul 14 19:22:04.185030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184902    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b1716d-9c75-49f0-a858-137465299a0a" containerName="revoke-keys"
Jul 14 19:22:04.185030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184968    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b1716d-9c75-49f0-a858-137465299a0a" containerName="revoke-keys"
Jul 14 19:22:04.185030 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.184985    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cbb0486-05f1-4158-bb42-bb7d1f7ff4eb" containerName="maas-api"
Jul 14 19:22:04.185915 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.185876    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:04.189363 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.189345    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.191784 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.191760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1877882c-serving-cert\""
Jul 14 19:22:04.191893 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.191813    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:22:04.192817 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.192800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:22:04.192917 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.192816    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:22:04.200628 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.200604    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"]
Jul 14 19:22:04.267822 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.267792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-maas-api-tls\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.267998 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.267853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2glr\" (UniqueName: \"kubernetes.io/projected/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-kube-api-access-h2glr\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.267998 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.267890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-openshift-service-ca\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.270089 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.270046    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:04.368693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.368604    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-openshift-service-ca\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.368856 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.368693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-maas-api-tls\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.368856 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.368765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2glr\" (UniqueName: \"kubernetes.io/projected/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-kube-api-access-h2glr\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.369376 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.369356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-openshift-service-ca\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.371257 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.371236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-maas-api-tls\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.378905 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.378880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2glr\" (UniqueName: \"kubernetes.io/projected/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-kube-api-access-h2glr\") pod \"maas-api-e2e-ait-1877882c-6b568658dd-96pkr\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.410517 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.410478    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:04.500838 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.500799    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:04.641581 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.641554    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"]
Jul 14 19:22:04.643294 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:22:04.643267    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060fd22f_d14d_4a7f_b99e_49b0d7a5bf14.slice/crio-ec079aabf76a771d378a9bf57680fd134fdeef1472e8c09b1b5325f868c3fa2d WatchSource:0}: Error finding container ec079aabf76a771d378a9bf57680fd134fdeef1472e8c09b1b5325f868c3fa2d: Status 404 returned error can't find the container with id ec079aabf76a771d378a9bf57680fd134fdeef1472e8c09b1b5325f868c3fa2d
Jul 14 19:22:04.819798 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.819764    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:04.949335 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:04.949247    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:05.127483 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:05.127446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr" event={"ID":"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14","Type":"ContainerStarted","Data":"ec079aabf76a771d378a9bf57680fd134fdeef1472e8c09b1b5325f868c3fa2d"}
Jul 14 19:22:06.133899 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:06.133854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr" event={"ID":"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14","Type":"ContainerStarted","Data":"d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f"}
Jul 14 19:22:06.134348 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:06.133969    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:06.176530 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:06.176474    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr" podStartSLOduration=1.627185388 podStartE2EDuration="2.176454191s" totalImagesPullingTime="549.268803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:06.174553487 +0000 UTC m=+1670.047111730" watchObservedRunningTime="2026-07-14 19:22:06.176454191 +0000 UTC m=+1670.049012413"
Jul 14 19:22:06.237544 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:06.237508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:09.135832 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.135796    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"]
Jul 14 19:22:09.138988 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.138971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:09.141629 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.141609    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:22:09.193102 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.193064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"]
Jul 14 19:22:09.213258 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.213227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a228a6d8-c974-4141-a389-7e07cab79250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1877882c-t46bf\" (UID: \"a228a6d8-c974-4141-a389-7e07cab79250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:09.257388 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.257357    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:09.314171 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.314132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a228a6d8-c974-4141-a389-7e07cab79250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1877882c-t46bf\" (UID: \"a228a6d8-c974-4141-a389-7e07cab79250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:09.314777 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.314756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a228a6d8-c974-4141-a389-7e07cab79250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1877882c-t46bf\" (UID: \"a228a6d8-c974-4141-a389-7e07cab79250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:09.357879 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.357837    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:09.450110 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.450027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:09.590940 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:09.590920    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"]
Jul 14 19:22:09.592518 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:22:09.592492    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda228a6d8_c974_4141_a389_7e07cab79250.slice/crio-3620d6a9e33eceaf21a3cc5a93461d50b6387408d7daf51a9232bf9e2c4c80bb WatchSource:0}: Error finding container 3620d6a9e33eceaf21a3cc5a93461d50b6387408d7daf51a9232bf9e2c4c80bb: Status 404 returned error can't find the container with id 3620d6a9e33eceaf21a3cc5a93461d50b6387408d7daf51a9232bf9e2c4c80bb
Jul 14 19:22:10.150248 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:10.150143    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:10.154003 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:10.153973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerStarted","Data":"90397c2fb0c023f66a28dba807f2f29f92fad2615c4ef99cb1a5ebb5b2601239"}
Jul 14 19:22:10.154143 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:10.154025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerStarted","Data":"3620d6a9e33eceaf21a3cc5a93461d50b6387408d7daf51a9232bf9e2c4c80bb"}
Jul 14 19:22:10.179941 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:10.179893    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" podStartSLOduration=1.179877177 podStartE2EDuration="1.179877177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:22:10.179758694 +0000 UTC m=+1674.052316915" watchObservedRunningTime="2026-07-14 19:22:10.179877177 +0000 UTC m=+1674.052435396"
Jul 14 19:22:10.249123 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:10.249078    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:22:11.160210 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:11.160167    2562 generic.go:356] "Generic (PLEG): container finished" podID="a228a6d8-c974-4141-a389-7e07cab79250" containerID="90397c2fb0c023f66a28dba807f2f29f92fad2615c4ef99cb1a5ebb5b2601239" exitCode=7
Jul 14 19:22:11.160624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:11.160264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerDied","Data":"90397c2fb0c023f66a28dba807f2f29f92fad2615c4ef99cb1a5ebb5b2601239"}
Jul 14 19:22:11.160624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:11.160464    2562 scope.go:117] "RemoveContainer" containerID="90397c2fb0c023f66a28dba807f2f29f92fad2615c4ef99cb1a5ebb5b2601239"
Jul 14 19:22:12.145256 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:12.145181    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:22:12.166612 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:12.166580    2562 generic.go:356] "Generic (PLEG): container finished" podID="a228a6d8-c974-4141-a389-7e07cab79250" containerID="a597c1851272ff166b949dc57063263b40176cd2077e856b379d201e91cc0a70" exitCode=7
Jul 14 19:22:12.166982 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:12.166666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerDied","Data":"a597c1851272ff166b949dc57063263b40176cd2077e856b379d201e91cc0a70"}
Jul 14 19:22:12.166982 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:12.166723    2562 scope.go:117] "RemoveContainer" containerID="90397c2fb0c023f66a28dba807f2f29f92fad2615c4ef99cb1a5ebb5b2601239"
Jul 14 19:22:12.166982 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:12.166936    2562 scope.go:117] "RemoveContainer" containerID="a597c1851272ff166b949dc57063263b40176cd2077e856b379d201e91cc0a70"
Jul 14 19:22:12.167166 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:22:12.167148    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-1877882c-t46bf_odh-ai-gateway-infra(a228a6d8-c974-4141-a389-7e07cab79250)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" podUID="a228a6d8-c974-4141-a389-7e07cab79250"
Jul 14 19:22:13.173747 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:13.173714    2562 scope.go:117] "RemoveContainer" containerID="a597c1851272ff166b949dc57063263b40176cd2077e856b379d201e91cc0a70"
Jul 14 19:22:13.174222 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:22:13.173988    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-1877882c-t46bf_odh-ai-gateway-infra(a228a6d8-c974-4141-a389-7e07cab79250)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" podUID="a228a6d8-c974-4141-a389-7e07cab79250"
Jul 14 19:22:24.616282 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:24.616246    2562 scope.go:117] "RemoveContainer" containerID="a597c1851272ff166b949dc57063263b40176cd2077e856b379d201e91cc0a70"
Jul 14 19:22:25.231986 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:25.231955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerStarted","Data":"cc5c531b3c698b1f192768e702d2773c53c547b810dcdc335b3af95d584084da"}
Jul 14 19:22:26.239822 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:26.239787    2562 generic.go:356] "Generic (PLEG): container finished" podID="a228a6d8-c974-4141-a389-7e07cab79250" containerID="cc5c531b3c698b1f192768e702d2773c53c547b810dcdc335b3af95d584084da" exitCode=0
Jul 14 19:22:26.240259 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:26.239844    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerDied","Data":"cc5c531b3c698b1f192768e702d2773c53c547b810dcdc335b3af95d584084da"}
Jul 14 19:22:26.240259 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:26.239895    2562 scope.go:117] "RemoveContainer" containerID="a597c1851272ff166b949dc57063263b40176cd2077e856b379d201e91cc0a70"
Jul 14 19:22:26.262785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:26.262748    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"]
Jul 14 19:22:27.382928 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:27.382903    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:27.485980 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:27.485945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a228a6d8-c974-4141-a389-7e07cab79250-maas-api-service-ca\") pod \"a228a6d8-c974-4141-a389-7e07cab79250\" (UID: \"a228a6d8-c974-4141-a389-7e07cab79250\") "
Jul 14 19:22:27.486361 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:27.486330    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a228a6d8-c974-4141-a389-7e07cab79250-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a228a6d8-c974-4141-a389-7e07cab79250" (UID: "a228a6d8-c974-4141-a389-7e07cab79250"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:22:27.586879 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:27.586793    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a228a6d8-c974-4141-a389-7e07cab79250-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:22:28.252362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:28.252329    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"
Jul 14 19:22:28.252558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:28.252375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf" event={"ID":"a228a6d8-c974-4141-a389-7e07cab79250","Type":"ContainerDied","Data":"3620d6a9e33eceaf21a3cc5a93461d50b6387408d7daf51a9232bf9e2c4c80bb"}
Jul 14 19:22:28.252558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:28.252447    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3620d6a9e33eceaf21a3cc5a93461d50b6387408d7daf51a9232bf9e2c4c80bb"
Jul 14 19:22:28.286758 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:28.286729    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"]
Jul 14 19:22:28.289918 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:28.289895    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-t46bf"]
Jul 14 19:22:28.624592 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:28.624496    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a228a6d8-c974-4141-a389-7e07cab79250" path="/var/lib/kubelet/pods/a228a6d8-c974-4141-a389-7e07cab79250/volumes"
Jul 14 19:22:59.179244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179163    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"]
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179515    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179526    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179546    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179551    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179558    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179564    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179644    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179652    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.179769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.179657    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a228a6d8-c974-4141-a389-7e07cab79250" containerName="revoke-keys"
Jul 14 19:22:59.182683 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.182666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:22:59.185043 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.185028    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:22:59.190816 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.190793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"]
Jul 14 19:22:59.274224 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.274183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dfc0f5-d267-41e5-8808-df916e07a249-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1877882c-k4spj\" (UID: \"c8dfc0f5-d267-41e5-8808-df916e07a249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:22:59.375081 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.375042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dfc0f5-d267-41e5-8808-df916e07a249-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1877882c-k4spj\" (UID: \"c8dfc0f5-d267-41e5-8808-df916e07a249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:22:59.375697 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.375676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dfc0f5-d267-41e5-8808-df916e07a249-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1877882c-k4spj\" (UID: \"c8dfc0f5-d267-41e5-8808-df916e07a249\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:22:59.494199 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.494165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:22:59.630366 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:22:59.630334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"]
Jul 14 19:22:59.632139 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:22:59.632099    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dfc0f5_d267_41e5_8808_df916e07a249.slice/crio-7c2daae051de998658098b6ca4e7fb8f8901bba9ce9ea26e7234e7667816823e WatchSource:0}: Error finding container 7c2daae051de998658098b6ca4e7fb8f8901bba9ce9ea26e7234e7667816823e: Status 404 returned error can't find the container with id 7c2daae051de998658098b6ca4e7fb8f8901bba9ce9ea26e7234e7667816823e
Jul 14 19:23:00.411254 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:00.411217    2562 generic.go:356] "Generic (PLEG): container finished" podID="c8dfc0f5-d267-41e5-8808-df916e07a249" containerID="fad955cf2a3320d751e4ed4c2a4c8cabae19907ddc57b0a1f070b1b0fa7ca2d0" exitCode=0
Jul 14 19:23:00.411631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:00.411301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj" event={"ID":"c8dfc0f5-d267-41e5-8808-df916e07a249","Type":"ContainerDied","Data":"fad955cf2a3320d751e4ed4c2a4c8cabae19907ddc57b0a1f070b1b0fa7ca2d0"}
Jul 14 19:23:00.411631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:00.411352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj" event={"ID":"c8dfc0f5-d267-41e5-8808-df916e07a249","Type":"ContainerStarted","Data":"7c2daae051de998658098b6ca4e7fb8f8901bba9ce9ea26e7234e7667816823e"}
Jul 14 19:23:01.547727 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:01.547703    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:23:01.696244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:01.696150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dfc0f5-d267-41e5-8808-df916e07a249-maas-api-service-ca\") pod \"c8dfc0f5-d267-41e5-8808-df916e07a249\" (UID: \"c8dfc0f5-d267-41e5-8808-df916e07a249\") "
Jul 14 19:23:01.696572 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:01.696549    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8dfc0f5-d267-41e5-8808-df916e07a249-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8dfc0f5-d267-41e5-8808-df916e07a249" (UID: "c8dfc0f5-d267-41e5-8808-df916e07a249"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:23:01.797614 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:01.797577    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dfc0f5-d267-41e5-8808-df916e07a249-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:23:02.421655 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:02.421625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj" event={"ID":"c8dfc0f5-d267-41e5-8808-df916e07a249","Type":"ContainerDied","Data":"7c2daae051de998658098b6ca4e7fb8f8901bba9ce9ea26e7234e7667816823e"}
Jul 14 19:23:02.421824 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:02.421662    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1877882c-k4spj"
Jul 14 19:23:02.421824 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:02.421674    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c2daae051de998658098b6ca4e7fb8f8901bba9ce9ea26e7234e7667816823e"
Jul 14 19:23:09.712943 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:09.712907    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"]
Jul 14 19:23:09.713454 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:09.713135    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr" podUID="060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" containerName="maas-api" containerID="cri-o://d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f" gracePeriod=30
Jul 14 19:23:09.745254 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:09.745221    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:09.955090 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:09.955059    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:23:10.082328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.082291    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-openshift-service-ca\") pod \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") "
Jul 14 19:23:10.082560 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.082349    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-maas-api-tls\") pod \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") "
Jul 14 19:23:10.082560 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.082368    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2glr\" (UniqueName: \"kubernetes.io/projected/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-kube-api-access-h2glr\") pod \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\" (UID: \"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14\") "
Jul 14 19:23:10.082785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.082760    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" (UID: "060fd22f-d14d-4a7f-b99e-49b0d7a5bf14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:23:10.084715 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.084681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" (UID: "060fd22f-d14d-4a7f-b99e-49b0d7a5bf14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:23:10.084715 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.084698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-kube-api-access-h2glr" (OuterVolumeSpecName: "kube-api-access-h2glr") pod "060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" (UID: "060fd22f-d14d-4a7f-b99e-49b0d7a5bf14"). InnerVolumeSpecName "kube-api-access-h2glr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:23:10.183845 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.183802    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:23:10.183845 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.183842    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:23:10.183845 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.183852    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2glr\" (UniqueName: \"kubernetes.io/projected/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14-kube-api-access-h2glr\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:23:10.461903 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.461811    2562 generic.go:356] "Generic (PLEG): container finished" podID="060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" containerID="d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f" exitCode=0
Jul 14 19:23:10.461903 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.461868    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"
Jul 14 19:23:10.462104 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.461897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr" event={"ID":"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14","Type":"ContainerDied","Data":"d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f"}
Jul 14 19:23:10.462104 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.461971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr" event={"ID":"060fd22f-d14d-4a7f-b99e-49b0d7a5bf14","Type":"ContainerDied","Data":"ec079aabf76a771d378a9bf57680fd134fdeef1472e8c09b1b5325f868c3fa2d"}
Jul 14 19:23:10.462104 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.461993    2562 scope.go:117] "RemoveContainer" containerID="d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f"
Jul 14 19:23:10.471182 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.471164    2562 scope.go:117] "RemoveContainer" containerID="d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f"
Jul 14 19:23:10.471570 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:23:10.471550    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f\": container with ID starting with d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f not found: ID does not exist" containerID="d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f"
Jul 14 19:23:10.471662 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.471579    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f"} err="failed to get container status \"d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f\": rpc error: code = NotFound desc = could not find container \"d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f\": container with ID starting with d0d9894c8a7cb98c46b01601df557fcfbc21b13036b6055ab5ab5e04b7aa886f not found: ID does not exist"
Jul 14 19:23:10.483044 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.483016    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"]
Jul 14 19:23:10.489769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.489717    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1877882c-6b568658dd-96pkr"]
Jul 14 19:23:10.621986 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:10.621953    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" path="/var/lib/kubelet/pods/060fd22f-d14d-4a7f-b99e-49b0d7a5bf14/volumes"
Jul 14 19:23:14.283505 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:14.283466    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:21.123287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:21.123252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:21.225661 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:21.225620    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:25.380966 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:25.380926    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:25.724371 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:25.724333    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:26.903423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.903361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:26.986244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986210    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"]
Jul 14 19:23:26.986639 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986618    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dfc0f5-d267-41e5-8808-df916e07a249" containerName="revoke-keys"
Jul 14 19:23:26.986639 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986641    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dfc0f5-d267-41e5-8808-df916e07a249" containerName="revoke-keys"
Jul 14 19:23:26.986801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986668    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" containerName="maas-api"
Jul 14 19:23:26.986801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986673    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" containerName="maas-api"
Jul 14 19:23:26.986801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986727    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8dfc0f5-d267-41e5-8808-df916e07a249" containerName="revoke-keys"
Jul 14 19:23:26.986801 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.986739    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="060fd22f-d14d-4a7f-b99e-49b0d7a5bf14" containerName="maas-api"
Jul 14 19:23:26.989831 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.989816    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:26.992649 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.992617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4961650e-serving-cert\""
Jul 14 19:23:26.992773 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.992671    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:23:26.993199 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.993177    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:26.993664 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.993651    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:23:26.993734 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:26.993671    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:23:27.001681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.001658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"]
Jul 14 19:23:27.045520 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.045491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.045520 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.045524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623f3e02-4dd8-48b3-8911-cee893574abf-openshift-service-ca\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.045749 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.045567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjfh2\" (UniqueName: \"kubernetes.io/projected/623f3e02-4dd8-48b3-8911-cee893574abf-kube-api-access-fjfh2\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.074375 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.074345    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:27.146776 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.146737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.146776 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.146775    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623f3e02-4dd8-48b3-8911-cee893574abf-openshift-service-ca\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.147000 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.146820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjfh2\" (UniqueName: \"kubernetes.io/projected/623f3e02-4dd8-48b3-8911-cee893574abf-kube-api-access-fjfh2\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.147000 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:23:27.146897    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-serving-cert: secret "maas-api-e2e-derive-4961650e-serving-cert" not found
Jul 14 19:23:27.147000 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:23:27.146967    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls podName:623f3e02-4dd8-48b3-8911-cee893574abf nodeName:}" failed. No retries permitted until 2026-07-14 19:23:27.646948362 +0000 UTC m=+1751.519506561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls") pod "maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" (UID: "623f3e02-4dd8-48b3-8911-cee893574abf") : secret "maas-api-e2e-derive-4961650e-serving-cert" not found
Jul 14 19:23:27.147449 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.147431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623f3e02-4dd8-48b3-8911-cee893574abf-openshift-service-ca\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.155478 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.155394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjfh2\" (UniqueName: \"kubernetes.io/projected/623f3e02-4dd8-48b3-8911-cee893574abf-kube-api-access-fjfh2\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.448996 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.448893    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:27.602394 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.602359    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:27.649818 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.649776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.652292 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.652270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls\") pod \"maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:27.697300 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.697271    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:27.901620 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:27.901572    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:28.045869 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:28.045842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"]
Jul 14 19:23:28.048137 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:23:28.048106    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623f3e02_4dd8_48b3_8911_cee893574abf.slice/crio-a1a281e1c9ff254af825c40fd1fc4a29e4c8345c988cdf603e7341c206cd8a90 WatchSource:0}: Error finding container a1a281e1c9ff254af825c40fd1fc4a29e4c8345c988cdf603e7341c206cd8a90: Status 404 returned error can't find the container with id a1a281e1c9ff254af825c40fd1fc4a29e4c8345c988cdf603e7341c206cd8a90
Jul 14 19:23:28.550198 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:28.550161    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" event={"ID":"623f3e02-4dd8-48b3-8911-cee893574abf","Type":"ContainerStarted","Data":"a1a281e1c9ff254af825c40fd1fc4a29e4c8345c988cdf603e7341c206cd8a90"}
Jul 14 19:23:29.556789 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:29.556754    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" event={"ID":"623f3e02-4dd8-48b3-8911-cee893574abf","Type":"ContainerStarted","Data":"6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4"}
Jul 14 19:23:29.557199 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:29.556872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:29.591521 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:29.591469    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" podStartSLOduration=3.134878516 podStartE2EDuration="3.591453305s" totalImagesPullingTime="456.574789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:23:29.589776967 +0000 UTC m=+1753.462335187" watchObservedRunningTime="2026-07-14 19:23:29.591453305 +0000 UTC m=+1753.464011525"
Jul 14 19:23:31.221126 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.221086    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"]
Jul 14 19:23:31.224654 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.224634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:31.227615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.227595    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:23:31.237247 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.237212    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"]
Jul 14 19:23:31.281585 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.281544    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25532ed9-1cdd-4b27-a251-4cdd2426d852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4961650e-4p7dp\" (UID: \"25532ed9-1cdd-4b27-a251-4cdd2426d852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:31.382870 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.382828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25532ed9-1cdd-4b27-a251-4cdd2426d852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4961650e-4p7dp\" (UID: \"25532ed9-1cdd-4b27-a251-4cdd2426d852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:31.383451 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.383435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25532ed9-1cdd-4b27-a251-4cdd2426d852-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4961650e-4p7dp\" (UID: \"25532ed9-1cdd-4b27-a251-4cdd2426d852\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:31.535067 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.534968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:31.674613 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:31.674588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"]
Jul 14 19:23:31.677201 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:23:31.677173    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25532ed9_1cdd_4b27_a251_4cdd2426d852.slice/crio-9071379f12c91b358b1a375e9cc2089e8ca9ee24e48c0f6b5cea7e533d346c9c WatchSource:0}: Error finding container 9071379f12c91b358b1a375e9cc2089e8ca9ee24e48c0f6b5cea7e533d346c9c: Status 404 returned error can't find the container with id 9071379f12c91b358b1a375e9cc2089e8ca9ee24e48c0f6b5cea7e533d346c9c
Jul 14 19:23:32.574193 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:32.574155    2562 generic.go:356] "Generic (PLEG): container finished" podID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerID="770936caa9981205f8194b3e8d03a40bebdc6684e01424a08468c909182c19ad" exitCode=7
Jul 14 19:23:32.574615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:32.574243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" event={"ID":"25532ed9-1cdd-4b27-a251-4cdd2426d852","Type":"ContainerDied","Data":"770936caa9981205f8194b3e8d03a40bebdc6684e01424a08468c909182c19ad"}
Jul 14 19:23:32.574615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:32.574297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" event={"ID":"25532ed9-1cdd-4b27-a251-4cdd2426d852","Type":"ContainerStarted","Data":"9071379f12c91b358b1a375e9cc2089e8ca9ee24e48c0f6b5cea7e533d346c9c"}
Jul 14 19:23:32.574615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:32.574447    2562 scope.go:117] "RemoveContainer" containerID="770936caa9981205f8194b3e8d03a40bebdc6684e01424a08468c909182c19ad"
Jul 14 19:23:32.841724 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:32.841682    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:32.920946 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:32.920914    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:33.581004 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:33.580921    2562 generic.go:356] "Generic (PLEG): container finished" podID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerID="c41661be5ae9d6b9a714c8de5a243cdc7f502bd72734c58f7dd8e5140c91f712" exitCode=7
Jul 14 19:23:33.581423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:33.581002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" event={"ID":"25532ed9-1cdd-4b27-a251-4cdd2426d852","Type":"ContainerDied","Data":"c41661be5ae9d6b9a714c8de5a243cdc7f502bd72734c58f7dd8e5140c91f712"}
Jul 14 19:23:33.581423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:33.581058    2562 scope.go:117] "RemoveContainer" containerID="770936caa9981205f8194b3e8d03a40bebdc6684e01424a08468c909182c19ad"
Jul 14 19:23:33.581423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:33.581272    2562 scope.go:117] "RemoveContainer" containerID="c41661be5ae9d6b9a714c8de5a243cdc7f502bd72734c58f7dd8e5140c91f712"
Jul 14 19:23:33.581608 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:23:33.581528    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4961650e-4p7dp_odh-ai-gateway-infra(25532ed9-1cdd-4b27-a251-4cdd2426d852)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852"
Jul 14 19:23:34.587239 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:34.587211    2562 scope.go:117] "RemoveContainer" containerID="c41661be5ae9d6b9a714c8de5a243cdc7f502bd72734c58f7dd8e5140c91f712"
Jul 14 19:23:34.587645 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:23:34.587445    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4961650e-4p7dp_odh-ai-gateway-infra(25532ed9-1cdd-4b27-a251-4cdd2426d852)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852"
Jul 14 19:23:35.567133 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:35.567099    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:23:36.531135 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:36.531106    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:23:48.615943 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:48.615906    2562 scope.go:117] "RemoveContainer" containerID="c41661be5ae9d6b9a714c8de5a243cdc7f502bd72734c58f7dd8e5140c91f712"
Jul 14 19:23:49.663290 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:49.663196    2562 generic.go:356] "Generic (PLEG): container finished" podID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerID="564df0730847e5708e00b7ebd4e0d3ad6cdfe3a21a6193827d2c5d676e7fc717" exitCode=0
Jul 14 19:23:49.663290 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:49.663277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" event={"ID":"25532ed9-1cdd-4b27-a251-4cdd2426d852","Type":"ContainerDied","Data":"564df0730847e5708e00b7ebd4e0d3ad6cdfe3a21a6193827d2c5d676e7fc717"}
Jul 14 19:23:49.663795 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:49.663348    2562 scope.go:117] "RemoveContainer" containerID="c41661be5ae9d6b9a714c8de5a243cdc7f502bd72734c58f7dd8e5140c91f712"
Jul 14 19:23:50.706740 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:50.706705    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"]
Jul 14 19:23:50.810896 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:50.810869    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:50.965974 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:50.965944    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25532ed9-1cdd-4b27-a251-4cdd2426d852-maas-api-service-ca\") pod \"25532ed9-1cdd-4b27-a251-4cdd2426d852\" (UID: \"25532ed9-1cdd-4b27-a251-4cdd2426d852\") "
Jul 14 19:23:50.966375 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:50.966354    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25532ed9-1cdd-4b27-a251-4cdd2426d852-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25532ed9-1cdd-4b27-a251-4cdd2426d852" (UID: "25532ed9-1cdd-4b27-a251-4cdd2426d852"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:23:51.066955 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:51.066921    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25532ed9-1cdd-4b27-a251-4cdd2426d852-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:23:51.675059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:51.675030    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"
Jul 14 19:23:51.675236 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:51.675030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp" event={"ID":"25532ed9-1cdd-4b27-a251-4cdd2426d852","Type":"ContainerDied","Data":"9071379f12c91b358b1a375e9cc2089e8ca9ee24e48c0f6b5cea7e533d346c9c"}
Jul 14 19:23:51.675236 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:51.675172    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9071379f12c91b358b1a375e9cc2089e8ca9ee24e48c0f6b5cea7e533d346c9c"
Jul 14 19:23:51.720816 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:51.720778    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"]
Jul 14 19:23:51.730157 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:51.730126    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-4p7dp"]
Jul 14 19:23:52.620910 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:23:52.620873    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" path="/var/lib/kubelet/pods/25532ed9-1cdd-4b27-a251-4cdd2426d852/volumes"
Jul 14 19:24:31.298652 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.298615    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"]
Jul 14 19:24:31.299163 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299134    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299163 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299154    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299168    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299177    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299198    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299208    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299515 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299295    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.299515 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.299314    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:24:31.302461 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.302441    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:31.305085 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.305068    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:24:31.312670 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.312645    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"]
Jul 14 19:24:31.440465 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.440392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49eed0a-e2ec-44f5-8e4b-754e85a49c14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4961650e-phfm7\" (UID: \"b49eed0a-e2ec-44f5-8e4b-754e85a49c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:31.541611 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.541562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49eed0a-e2ec-44f5-8e4b-754e85a49c14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4961650e-phfm7\" (UID: \"b49eed0a-e2ec-44f5-8e4b-754e85a49c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:31.542203 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.542182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49eed0a-e2ec-44f5-8e4b-754e85a49c14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4961650e-phfm7\" (UID: \"b49eed0a-e2ec-44f5-8e4b-754e85a49c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:31.613832 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.613755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:31.758532 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.758504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"]
Jul 14 19:24:31.760771 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:24:31.760735    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49eed0a_e2ec_44f5_8e4b_754e85a49c14.slice/crio-101c970e277cdb8b2512f7b40d675169578161226ab56f96fabab12695560cec WatchSource:0}: Error finding container 101c970e277cdb8b2512f7b40d675169578161226ab56f96fabab12695560cec: Status 404 returned error can't find the container with id 101c970e277cdb8b2512f7b40d675169578161226ab56f96fabab12695560cec
Jul 14 19:24:31.868093 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:31.868002    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7" event={"ID":"b49eed0a-e2ec-44f5-8e4b-754e85a49c14","Type":"ContainerStarted","Data":"101c970e277cdb8b2512f7b40d675169578161226ab56f96fabab12695560cec"}
Jul 14 19:24:32.874187 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:32.874156    2562 generic.go:356] "Generic (PLEG): container finished" podID="b49eed0a-e2ec-44f5-8e4b-754e85a49c14" containerID="671cd2ae33462151b8d8c6db2db178a8010e1488388d71942034ce64b48d2f34" exitCode=0
Jul 14 19:24:32.874187 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:32.874191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7" event={"ID":"b49eed0a-e2ec-44f5-8e4b-754e85a49c14","Type":"ContainerDied","Data":"671cd2ae33462151b8d8c6db2db178a8010e1488388d71942034ce64b48d2f34"}
Jul 14 19:24:34.008588 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.008565    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:34.064810 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.064770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49eed0a-e2ec-44f5-8e4b-754e85a49c14-maas-api-service-ca\") pod \"b49eed0a-e2ec-44f5-8e4b-754e85a49c14\" (UID: \"b49eed0a-e2ec-44f5-8e4b-754e85a49c14\") "
Jul 14 19:24:34.065121 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.065098    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49eed0a-e2ec-44f5-8e4b-754e85a49c14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b49eed0a-e2ec-44f5-8e4b-754e85a49c14" (UID: "b49eed0a-e2ec-44f5-8e4b-754e85a49c14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:24:34.165825 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.165733    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b49eed0a-e2ec-44f5-8e4b-754e85a49c14-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:24:34.884864 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.884770    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7" event={"ID":"b49eed0a-e2ec-44f5-8e4b-754e85a49c14","Type":"ContainerDied","Data":"101c970e277cdb8b2512f7b40d675169578161226ab56f96fabab12695560cec"}
Jul 14 19:24:34.884864 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.884789    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4961650e-phfm7"
Jul 14 19:24:34.884864 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:34.884819    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="101c970e277cdb8b2512f7b40d675169578161226ab56f96fabab12695560cec"
Jul 14 19:24:41.850995 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:41.850959    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"]
Jul 14 19:24:41.851515 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:41.851203    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" podUID="623f3e02-4dd8-48b3-8911-cee893574abf" containerName="maas-api" containerID="cri-o://6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4" gracePeriod=30
Jul 14 19:24:41.882749 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:41.882719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:42.096664 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.096637    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:24:42.135058 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.134974    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjfh2\" (UniqueName: \"kubernetes.io/projected/623f3e02-4dd8-48b3-8911-cee893574abf-kube-api-access-fjfh2\") pod \"623f3e02-4dd8-48b3-8911-cee893574abf\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") "
Jul 14 19:24:42.135058 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.135012    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls\") pod \"623f3e02-4dd8-48b3-8911-cee893574abf\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") "
Jul 14 19:24:42.135058 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.135029    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623f3e02-4dd8-48b3-8911-cee893574abf-openshift-service-ca\") pod \"623f3e02-4dd8-48b3-8911-cee893574abf\" (UID: \"623f3e02-4dd8-48b3-8911-cee893574abf\") "
Jul 14 19:24:42.135499 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.135471    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623f3e02-4dd8-48b3-8911-cee893574abf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "623f3e02-4dd8-48b3-8911-cee893574abf" (UID: "623f3e02-4dd8-48b3-8911-cee893574abf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:24:42.137301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.137272    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "623f3e02-4dd8-48b3-8911-cee893574abf" (UID: "623f3e02-4dd8-48b3-8911-cee893574abf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:24:42.137394 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.137373    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623f3e02-4dd8-48b3-8911-cee893574abf-kube-api-access-fjfh2" (OuterVolumeSpecName: "kube-api-access-fjfh2") pod "623f3e02-4dd8-48b3-8911-cee893574abf" (UID: "623f3e02-4dd8-48b3-8911-cee893574abf"). InnerVolumeSpecName "kube-api-access-fjfh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:24:42.235617 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.235582    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjfh2\" (UniqueName: \"kubernetes.io/projected/623f3e02-4dd8-48b3-8911-cee893574abf-kube-api-access-fjfh2\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:24:42.235617 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.235611    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/623f3e02-4dd8-48b3-8911-cee893574abf-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:24:42.235617 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.235623    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/623f3e02-4dd8-48b3-8911-cee893574abf-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:24:42.922222 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.922129    2562 generic.go:356] "Generic (PLEG): container finished" podID="623f3e02-4dd8-48b3-8911-cee893574abf" containerID="6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4" exitCode=0
Jul 14 19:24:42.922222 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.922170    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"
Jul 14 19:24:42.922754 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.922218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" event={"ID":"623f3e02-4dd8-48b3-8911-cee893574abf","Type":"ContainerDied","Data":"6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4"}
Jul 14 19:24:42.922754 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.922273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4" event={"ID":"623f3e02-4dd8-48b3-8911-cee893574abf","Type":"ContainerDied","Data":"a1a281e1c9ff254af825c40fd1fc4a29e4c8345c988cdf603e7341c206cd8a90"}
Jul 14 19:24:42.922754 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.922302    2562 scope.go:117] "RemoveContainer" containerID="6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4"
Jul 14 19:24:42.931504 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.931486    2562 scope.go:117] "RemoveContainer" containerID="6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4"
Jul 14 19:24:42.931795 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:24:42.931772    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4\": container with ID starting with 6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4 not found: ID does not exist" containerID="6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4"
Jul 14 19:24:42.931898 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.931804    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4"} err="failed to get container status \"6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4\": rpc error: code = NotFound desc = could not find container \"6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4\": container with ID starting with 6e06b9225bed57f6fe659b8b132128f1aac71bf51465591b8126415988a1b5a4 not found: ID does not exist"
Jul 14 19:24:42.940282 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.940256    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"]
Jul 14 19:24:42.943807 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:42.943783    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4961650e-88d8fc7c7-jlbs4"]
Jul 14 19:24:44.621757 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:44.621724    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623f3e02-4dd8-48b3-8911-cee893574abf" path="/var/lib/kubelet/pods/623f3e02-4dd8-48b3-8911-cee893574abf/volumes"
Jul 14 19:24:46.387592 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:46.387543    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:52.247530 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:52.247454    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:52.316883 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:52.316851    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:56.587124 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:56.587082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:57.162838 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:57.162786    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:57.260607 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:57.260565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:57.361305 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:57.361269    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:24:58.826305 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:24:58.826266    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:03.021502 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:03.021252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:04.894335 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:04.894295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:05.003511 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:05.003476    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:05.093034 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:05.092995    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:05.194220 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:05.194127    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:05.633190 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:05.633139    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:12.446506 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:12.446467    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:31.623197 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:31.623156    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:37.633537 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:37.633448    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:37.706226 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:37.706186    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:37.800963 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:37.800919    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:37.902806 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:37.902716    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:53.720608 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:53.720567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:59.771063 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:59.771019    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:59.877611 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:59.877574    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:25:59.970280 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:25:59.970245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:13.311821 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:13.311778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:13.412631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:13.412586    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:16.728971 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:16.728944    2562 scope.go:117] "RemoveContainer" containerID="6e422d3f056ff5707febb031ed7dd408757495eac0829c6bc0c8cbebfc54b8c5"
Jul 14 19:26:17.349950 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:17.349908    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:17.459986 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:17.459943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:17.552036 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:17.551969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:17.653079 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:17.652974    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:18.423635 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:18.423599    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:18.506046 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:18.505993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:18.606378 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:18.606346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:20.206761 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:20.206717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:26.348492 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:26.348459    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:26.449233 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:26.449195    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:27.018277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:27.018244    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:33.416563 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:33.416524    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:33.517029 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:33.516993    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:33.784555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:33.784512    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:34.209745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:34.209642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:34.486457 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:34.486386    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:37.755241 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.755210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:37.840137 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840108    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"]
Jul 14 19:26:37.840480 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840467    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623f3e02-4dd8-48b3-8911-cee893574abf" containerName="maas-api"
Jul 14 19:26:37.840532 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840482    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="623f3e02-4dd8-48b3-8911-cee893574abf" containerName="maas-api"
Jul 14 19:26:37.840532 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840498    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b49eed0a-e2ec-44f5-8e4b-754e85a49c14" containerName="revoke-keys"
Jul 14 19:26:37.840532 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840504    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b49eed0a-e2ec-44f5-8e4b-754e85a49c14" containerName="revoke-keys"
Jul 14 19:26:37.840630 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840561    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="623f3e02-4dd8-48b3-8911-cee893574abf" containerName="maas-api"
Jul 14 19:26:37.840630 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840571    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25532ed9-1cdd-4b27-a251-4cdd2426d852" containerName="revoke-keys"
Jul 14 19:26:37.840630 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.840578    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b49eed0a-e2ec-44f5-8e4b-754e85a49c14" containerName="revoke-keys"
Jul 14 19:26:37.843476 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.843454    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:37.846969 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.846930    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-393716c6-serving-cert\""
Jul 14 19:26:37.846969 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.846967    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:26:37.847531 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.847512    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:26:37.847639 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.847512    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:26:37.857328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.857305    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"]
Jul 14 19:26:37.862449 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.862422    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:37.953542 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.953503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k524h\" (UniqueName: \"kubernetes.io/projected/127b329b-07cb-4944-ac06-ecfb9236da48-kube-api-access-k524h\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:37.953710 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.953581    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127b329b-07cb-4944-ac06-ecfb9236da48-maas-api-tls\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:37.953710 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.953618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127b329b-07cb-4944-ac06-ecfb9236da48-openshift-service-ca\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:37.956994 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:37.956964    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:38.054970 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.054869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127b329b-07cb-4944-ac06-ecfb9236da48-maas-api-tls\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.054970 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.054927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127b329b-07cb-4944-ac06-ecfb9236da48-openshift-service-ca\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.054970 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.054963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k524h\" (UniqueName: \"kubernetes.io/projected/127b329b-07cb-4944-ac06-ecfb9236da48-kube-api-access-k524h\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.055759 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.055731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127b329b-07cb-4944-ac06-ecfb9236da48-openshift-service-ca\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.056211 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.056187    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:38.057792 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.057766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127b329b-07cb-4944-ac06-ecfb9236da48-maas-api-tls\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.065669 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.065645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k524h\" (UniqueName: \"kubernetes.io/projected/127b329b-07cb-4944-ac06-ecfb9236da48-kube-api-access-k524h\") pod \"maas-api-e2e-mt-393716c6-677844b659-6z4z8\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.154319 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.154271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:38.295769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.295747    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"]
Jul 14 19:26:38.298068 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:26:38.298041    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127b329b_07cb_4944_ac06_ecfb9236da48.slice/crio-447f49e2845431da0164e3fd4c8c69ab12c2f2854b62982d8dba091d0f43826c WatchSource:0}: Error finding container 447f49e2845431da0164e3fd4c8c69ab12c2f2854b62982d8dba091d0f43826c: Status 404 returned error can't find the container with id 447f49e2845431da0164e3fd4c8c69ab12c2f2854b62982d8dba091d0f43826c
Jul 14 19:26:38.299820 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.299803    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:26:38.471092 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.471056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:38.475394 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.475364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8" event={"ID":"127b329b-07cb-4944-ac06-ecfb9236da48","Type":"ContainerStarted","Data":"447f49e2845431da0164e3fd4c8c69ab12c2f2854b62982d8dba091d0f43826c"}
Jul 14 19:26:38.567939 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:38.567906    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:42.237234 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:42.237196    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:42.496470 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:42.496360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8" event={"ID":"127b329b-07cb-4944-ac06-ecfb9236da48","Type":"ContainerStarted","Data":"8c9f0600021757b94bd4094a4faaf0aa63a6b7f0385df4bbe348199577d7da1c"}
Jul 14 19:26:42.496620 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:42.496524    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:42.518575 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:42.518524    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8" podStartSLOduration=2.176882221 podStartE2EDuration="5.518506853s" totalImagesPullingTime="3.341624632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:26:42.516303609 +0000 UTC m=+1946.388861829" watchObservedRunningTime="2026-07-14 19:26:42.518506853 +0000 UTC m=+1946.391065104"
Jul 14 19:26:42.593585 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:42.593548    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:42.903595 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:42.903508    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:43.006604 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:43.006567    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:43.328880 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:43.328828    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:48.508213 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:48.508187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:26:49.518679 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:49.518642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:49.907005 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:49.906924    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"]
Jul 14 19:26:49.910107 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:49.910085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:49.912620 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:49.912601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:26:49.919191 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:49.919164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"]
Jul 14 19:26:50.064807 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:50.064765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c022d2f-a9eb-474e-a117-66fc66b8b126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-393716c6-skmsk\" (UID: \"1c022d2f-a9eb-474e-a117-66fc66b8b126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:50.166359 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:50.166270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c022d2f-a9eb-474e-a117-66fc66b8b126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-393716c6-skmsk\" (UID: \"1c022d2f-a9eb-474e-a117-66fc66b8b126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:50.166954 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:50.166935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c022d2f-a9eb-474e-a117-66fc66b8b126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-393716c6-skmsk\" (UID: \"1c022d2f-a9eb-474e-a117-66fc66b8b126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:50.221477 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:50.221437    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:50.359894 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:50.359870    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"]
Jul 14 19:26:50.361979 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:26:50.361952    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c022d2f_a9eb_474e_a117_66fc66b8b126.slice/crio-7ae4ad6bcb11ac4f78c5a9151d9c9931dbfaf282e909464916907181ff24d512 WatchSource:0}: Error finding container 7ae4ad6bcb11ac4f78c5a9151d9c9931dbfaf282e909464916907181ff24d512: Status 404 returned error can't find the container with id 7ae4ad6bcb11ac4f78c5a9151d9c9931dbfaf282e909464916907181ff24d512
Jul 14 19:26:50.536545 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:50.536508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk" event={"ID":"1c022d2f-a9eb-474e-a117-66fc66b8b126","Type":"ContainerStarted","Data":"7ae4ad6bcb11ac4f78c5a9151d9c9931dbfaf282e909464916907181ff24d512"}
Jul 14 19:26:51.543304 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:51.543269    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c022d2f-a9eb-474e-a117-66fc66b8b126" containerID="493d09ab774110ce00d48a9de9372d6cefff48d34ce11b453ccb449b6a9dd23d" exitCode=0
Jul 14 19:26:51.543772 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:51.543329    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk" event={"ID":"1c022d2f-a9eb-474e-a117-66fc66b8b126","Type":"ContainerDied","Data":"493d09ab774110ce00d48a9de9372d6cefff48d34ce11b453ccb449b6a9dd23d"}
Jul 14 19:26:52.473144 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:52.473099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:26:52.677739 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:52.677706    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:52.790058 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:52.789966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c022d2f-a9eb-474e-a117-66fc66b8b126-maas-api-service-ca\") pod \"1c022d2f-a9eb-474e-a117-66fc66b8b126\" (UID: \"1c022d2f-a9eb-474e-a117-66fc66b8b126\") "
Jul 14 19:26:52.790339 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:52.790317    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c022d2f-a9eb-474e-a117-66fc66b8b126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c022d2f-a9eb-474e-a117-66fc66b8b126" (UID: "1c022d2f-a9eb-474e-a117-66fc66b8b126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:26:52.891481 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:52.891438    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c022d2f-a9eb-474e-a117-66fc66b8b126-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:26:53.553522 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:53.553486    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk"
Jul 14 19:26:53.553522 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:53.553514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-393716c6-skmsk" event={"ID":"1c022d2f-a9eb-474e-a117-66fc66b8b126","Type":"ContainerDied","Data":"7ae4ad6bcb11ac4f78c5a9151d9c9931dbfaf282e909464916907181ff24d512"}
Jul 14 19:26:53.553747 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:26:53.553568    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae4ad6bcb11ac4f78c5a9151d9c9931dbfaf282e909464916907181ff24d512"
Jul 14 19:27:00.364282 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.364249    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"]
Jul 14 19:27:00.364787 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.364517    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8" podUID="127b329b-07cb-4944-ac06-ecfb9236da48" containerName="maas-api" containerID="cri-o://8c9f0600021757b94bd4094a4faaf0aa63a6b7f0385df4bbe348199577d7da1c" gracePeriod=30
Jul 14 19:27:00.388556 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.388510    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:00.587726 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.587695    2562 generic.go:356] "Generic (PLEG): container finished" podID="127b329b-07cb-4944-ac06-ecfb9236da48" containerID="8c9f0600021757b94bd4094a4faaf0aa63a6b7f0385df4bbe348199577d7da1c" exitCode=0
Jul 14 19:27:00.587853 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.587772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8" event={"ID":"127b329b-07cb-4944-ac06-ecfb9236da48","Type":"ContainerDied","Data":"8c9f0600021757b94bd4094a4faaf0aa63a6b7f0385df4bbe348199577d7da1c"}
Jul 14 19:27:00.603896 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.603871    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:27:00.761914 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.761879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127b329b-07cb-4944-ac06-ecfb9236da48-maas-api-tls\") pod \"127b329b-07cb-4944-ac06-ecfb9236da48\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") "
Jul 14 19:27:00.762105 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.761933    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k524h\" (UniqueName: \"kubernetes.io/projected/127b329b-07cb-4944-ac06-ecfb9236da48-kube-api-access-k524h\") pod \"127b329b-07cb-4944-ac06-ecfb9236da48\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") "
Jul 14 19:27:00.762105 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.761999    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127b329b-07cb-4944-ac06-ecfb9236da48-openshift-service-ca\") pod \"127b329b-07cb-4944-ac06-ecfb9236da48\" (UID: \"127b329b-07cb-4944-ac06-ecfb9236da48\") "
Jul 14 19:27:00.762491 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.762463    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127b329b-07cb-4944-ac06-ecfb9236da48-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "127b329b-07cb-4944-ac06-ecfb9236da48" (UID: "127b329b-07cb-4944-ac06-ecfb9236da48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:27:00.764257 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.764225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127b329b-07cb-4944-ac06-ecfb9236da48-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "127b329b-07cb-4944-ac06-ecfb9236da48" (UID: "127b329b-07cb-4944-ac06-ecfb9236da48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:27:00.764354 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.764257    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127b329b-07cb-4944-ac06-ecfb9236da48-kube-api-access-k524h" (OuterVolumeSpecName: "kube-api-access-k524h") pod "127b329b-07cb-4944-ac06-ecfb9236da48" (UID: "127b329b-07cb-4944-ac06-ecfb9236da48"). InnerVolumeSpecName "kube-api-access-k524h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:27:00.862906 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.862866    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/127b329b-07cb-4944-ac06-ecfb9236da48-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:27:00.862906 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.862900    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k524h\" (UniqueName: \"kubernetes.io/projected/127b329b-07cb-4944-ac06-ecfb9236da48-kube-api-access-k524h\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:27:00.862906 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:00.862912    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/127b329b-07cb-4944-ac06-ecfb9236da48-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:27:01.594079 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:01.594042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8" event={"ID":"127b329b-07cb-4944-ac06-ecfb9236da48","Type":"ContainerDied","Data":"447f49e2845431da0164e3fd4c8c69ab12c2f2854b62982d8dba091d0f43826c"}
Jul 14 19:27:01.594580 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:01.594108    2562 scope.go:117] "RemoveContainer" containerID="8c9f0600021757b94bd4094a4faaf0aa63a6b7f0385df4bbe348199577d7da1c"
Jul 14 19:27:01.594580 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:01.594056    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"
Jul 14 19:27:01.615731 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:01.615699    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"]
Jul 14 19:27:01.619569 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:01.619539    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-393716c6-677844b659-6z4z8"]
Jul 14 19:27:02.621009 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:02.620978    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="127b329b-07cb-4944-ac06-ecfb9236da48" path="/var/lib/kubelet/pods/127b329b-07cb-4944-ac06-ecfb9236da48/volumes"
Jul 14 19:27:04.998948 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:04.998904    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:10.201838 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:10.201750    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:10.302649 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:10.302618    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:11.436086 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:11.436051    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:11.536602 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:11.536565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:15.664479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:15.664446    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:15.900423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:15.900361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:16.289991 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:16.289954    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:16.358295 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:16.358261    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:16.747082 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:16.747050    2562 scope.go:117] "RemoveContainer" containerID="8bc6ede383a62ffaf60eac6338f1d7e632cb316a1e8813e802a94603f5464a34"
Jul 14 19:27:22.179318 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:22.179284    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:22.286753 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:22.286709    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:22.380010 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:22.379967    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:22.556068 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:22.556017    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:26.476754 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:26.476717    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:28.587537 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:28.587497    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:28.656050 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:28.656013    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:35.397043 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:35.397003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:35.724277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:35.724240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:35.912134 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:35.912098    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:38.852328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.852290    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:38.997039 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997002    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"]
Jul 14 19:27:38.997452 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997437    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="127b329b-07cb-4944-ac06-ecfb9236da48" containerName="maas-api"
Jul 14 19:27:38.997507 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997453    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="127b329b-07cb-4944-ac06-ecfb9236da48" containerName="maas-api"
Jul 14 19:27:38.997507 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997483    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c022d2f-a9eb-474e-a117-66fc66b8b126" containerName="revoke-keys"
Jul 14 19:27:38.997507 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997490    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c022d2f-a9eb-474e-a117-66fc66b8b126" containerName="revoke-keys"
Jul 14 19:27:38.997606 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997548    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c022d2f-a9eb-474e-a117-66fc66b8b126" containerName="revoke-keys"
Jul 14 19:27:38.997606 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:38.997561    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="127b329b-07cb-4944-ac06-ecfb9236da48" containerName="maas-api"
Jul 14 19:27:39.000693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.000672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.005576 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.005550    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-618aef4e-serving-cert\""
Jul 14 19:27:39.005710 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.005576    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:27:39.005710 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.005617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:27:39.006595 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.006576    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:27:39.027109 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.027071    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"]
Jul 14 19:27:39.075274 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.075233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:39.114645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.114538    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805f1ea2-2a72-4abd-a100-fe217a03f97f-openshift-service-ca\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.114820 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.114689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wtv\" (UniqueName: \"kubernetes.io/projected/805f1ea2-2a72-4abd-a100-fe217a03f97f-kube-api-access-77wtv\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.114820 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.114724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805f1ea2-2a72-4abd-a100-fe217a03f97f-maas-api-tls\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.179590 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.179551    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:39.215580 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.215548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wtv\" (UniqueName: \"kubernetes.io/projected/805f1ea2-2a72-4abd-a100-fe217a03f97f-kube-api-access-77wtv\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.215580 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.215587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805f1ea2-2a72-4abd-a100-fe217a03f97f-maas-api-tls\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.215857 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.215619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805f1ea2-2a72-4abd-a100-fe217a03f97f-openshift-service-ca\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.216359 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.216333    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805f1ea2-2a72-4abd-a100-fe217a03f97f-openshift-service-ca\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.218201 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.218177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805f1ea2-2a72-4abd-a100-fe217a03f97f-maas-api-tls\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.230573 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.230547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wtv\" (UniqueName: \"kubernetes.io/projected/805f1ea2-2a72-4abd-a100-fe217a03f97f-kube-api-access-77wtv\") pod \"maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.284349 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.284311    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:39.311125 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.311094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:39.453694 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.453670    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"]
Jul 14 19:27:39.455589 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:27:39.455556    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805f1ea2_2a72_4abd_a100_fe217a03f97f.slice/crio-dc6c1322b7e6be4baa4832812666bc81be17ac5827d508151d6b60301e9837ac WatchSource:0}: Error finding container dc6c1322b7e6be4baa4832812666bc81be17ac5827d508151d6b60301e9837ac: Status 404 returned error can't find the container with id dc6c1322b7e6be4baa4832812666bc81be17ac5827d508151d6b60301e9837ac
Jul 14 19:27:39.558742 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.558691    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:39.657819 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.657725    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:39.778755 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:39.778725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q" event={"ID":"805f1ea2-2a72-4abd-a100-fe217a03f97f","Type":"ContainerStarted","Data":"dc6c1322b7e6be4baa4832812666bc81be17ac5827d508151d6b60301e9837ac"}
Jul 14 19:27:40.784664 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:40.784623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q" event={"ID":"805f1ea2-2a72-4abd-a100-fe217a03f97f","Type":"ContainerStarted","Data":"d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8"}
Jul 14 19:27:40.785055 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:40.784710    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:40.805379 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:40.805329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q" podStartSLOduration=2.471894142 podStartE2EDuration="2.805312936s" totalImagesPullingTime="333.418794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:27:40.802517939 +0000 UTC m=+2004.675076162" watchObservedRunningTime="2026-07-14 19:27:40.805312936 +0000 UTC m=+2004.677871155"
Jul 14 19:27:43.316980 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:43.316940    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:46.794696 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:46.794669    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:27:49.347307 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:49.347240    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:52.768826 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.768796    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"]
Jul 14 19:27:52.772055 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.772036    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.774822 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.774800    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-618aef4e\"/\"kube-root-ca.crt\""
Jul 14 19:27:52.775858 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.775828    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-618aef4e\"/\"openshift-service-ca.crt\""
Jul 14 19:27:52.775858 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.775846    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-618aef4e\"/\"e2e-lifecycle-model-618aef4e-kserve-self-signed-certs\""
Jul 14 19:27:52.776026 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.776001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-618aef4e\"/\"default-dockercfg-f9zpf\""
Jul 14 19:27:52.782036 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.782010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"]
Jul 14 19:27:52.810188 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.810137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:27:52.840606 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.840574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tmp-dir\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.840821 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.840625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tls-certs\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.840821 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.840654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-home\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.840821 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.840695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62mr\" (UniqueName: \"kubernetes.io/projected/2e2c5d53-ef62-43c2-9b37-df465076aa9e-kube-api-access-q62mr\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.840821 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.840727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-model-cache\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.840821 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.840805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-dshm\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942016 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.941976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-dshm\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942227 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tmp-dir\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942227 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tls-certs\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942227 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-home\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942227 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62mr\" (UniqueName: \"kubernetes.io/projected/2e2c5d53-ef62-43c2-9b37-df465076aa9e-kube-api-access-q62mr\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942227 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-model-cache\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942565 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tmp-dir\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942650 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-home\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.942650 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.942622    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-model-cache\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.944531 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.944510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-dshm\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.944715 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.944697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tls-certs\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:52.950979 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:52.950953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62mr\" (UniqueName: \"kubernetes.io/projected/2e2c5d53-ef62-43c2-9b37-df465076aa9e-kube-api-access-q62mr\") pod \"e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") " pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:53.083392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:53.083285    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:27:53.219007 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:53.218984    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"]
Jul 14 19:27:53.220744 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:27:53.220707    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e2c5d53_ef62_43c2_9b37_df465076aa9e.slice/crio-e18b62676319c7e2a9f36ad439e5a1e77e84343282bb39dd13952b058dadced9 WatchSource:0}: Error finding container e18b62676319c7e2a9f36ad439e5a1e77e84343282bb39dd13952b058dadced9: Status 404 returned error can't find the container with id e18b62676319c7e2a9f36ad439e5a1e77e84343282bb39dd13952b058dadced9
Jul 14 19:27:53.846720 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:53.846681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf" event={"ID":"2e2c5d53-ef62-43c2-9b37-df465076aa9e","Type":"ContainerStarted","Data":"a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae"}
Jul 14 19:27:53.847104 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:27:53.846727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf" event={"ID":"2e2c5d53-ef62-43c2-9b37-df465076aa9e","Type":"ContainerStarted","Data":"e18b62676319c7e2a9f36ad439e5a1e77e84343282bb39dd13952b058dadced9"}
Jul 14 19:28:03.084476 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:03.084435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:28:03.084950 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:03.084493    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:28:03.096761 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:03.096725    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:28:03.116562 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:03.116515    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf" podStartSLOduration=11.116497845 podStartE2EDuration="11.116497845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:27:53.864179886 +0000 UTC m=+2017.736738106" watchObservedRunningTime="2026-07-14 19:28:03.116497845 +0000 UTC m=+2026.989056435"
Jul 14 19:28:03.911847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:03.911818    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:28:16.376653 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:16.376611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:28:16.477396 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:16.477361    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:28:21.556211 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.556179    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"]
Jul 14 19:28:21.559613 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.559595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:21.562297 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.562272    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:28:21.569731 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.569698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"]
Jul 14 19:28:21.709848 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.709795    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78ecefb-8dbb-4229-9778-b42b817a6be9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-618aef4e-2h5td\" (UID: \"c78ecefb-8dbb-4229-9778-b42b817a6be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:21.810529 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.810440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78ecefb-8dbb-4229-9778-b42b817a6be9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-618aef4e-2h5td\" (UID: \"c78ecefb-8dbb-4229-9778-b42b817a6be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:21.811075 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.811055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78ecefb-8dbb-4229-9778-b42b817a6be9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-618aef4e-2h5td\" (UID: \"c78ecefb-8dbb-4229-9778-b42b817a6be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:21.871459 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:21.871425    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:22.006288 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:22.006269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"]
Jul 14 19:28:22.009132 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:28:22.009100    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78ecefb_8dbb_4229_9778_b42b817a6be9.slice/crio-cd11de87ee7cee38ca5fd7b968b0eda992edd029217d79f404644b69adb3a973 WatchSource:0}: Error finding container cd11de87ee7cee38ca5fd7b968b0eda992edd029217d79f404644b69adb3a973: Status 404 returned error can't find the container with id cd11de87ee7cee38ca5fd7b968b0eda992edd029217d79f404644b69adb3a973
Jul 14 19:28:22.997059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:22.996966    2562 generic.go:356] "Generic (PLEG): container finished" podID="c78ecefb-8dbb-4229-9778-b42b817a6be9" containerID="2b41595c311b0b3dadc0a0ee3e356ce7ffdeea49ccef9a5b0375d840277950e0" exitCode=0
Jul 14 19:28:22.997059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:22.997047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td" event={"ID":"c78ecefb-8dbb-4229-9778-b42b817a6be9","Type":"ContainerDied","Data":"2b41595c311b0b3dadc0a0ee3e356ce7ffdeea49ccef9a5b0375d840277950e0"}
Jul 14 19:28:22.997479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:22.997121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td" event={"ID":"c78ecefb-8dbb-4229-9778-b42b817a6be9","Type":"ContainerStarted","Data":"cd11de87ee7cee38ca5fd7b968b0eda992edd029217d79f404644b69adb3a973"}
Jul 14 19:28:24.132716 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:24.132689    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:24.232472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:24.232435    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78ecefb-8dbb-4229-9778-b42b817a6be9-maas-api-service-ca\") pod \"c78ecefb-8dbb-4229-9778-b42b817a6be9\" (UID: \"c78ecefb-8dbb-4229-9778-b42b817a6be9\") "
Jul 14 19:28:24.232842 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:24.232815    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78ecefb-8dbb-4229-9778-b42b817a6be9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c78ecefb-8dbb-4229-9778-b42b817a6be9" (UID: "c78ecefb-8dbb-4229-9778-b42b817a6be9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:28:24.333583 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:24.333489    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c78ecefb-8dbb-4229-9778-b42b817a6be9-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:25.008500 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:25.008465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td" event={"ID":"c78ecefb-8dbb-4229-9778-b42b817a6be9","Type":"ContainerDied","Data":"cd11de87ee7cee38ca5fd7b968b0eda992edd029217d79f404644b69adb3a973"}
Jul 14 19:28:25.008500 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:25.008510    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-618aef4e-2h5td"
Jul 14 19:28:25.008716 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:25.008521    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd11de87ee7cee38ca5fd7b968b0eda992edd029217d79f404644b69adb3a973"
Jul 14 19:28:32.059832 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.059799    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"]
Jul 14 19:28:32.060371 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.060081    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q" podUID="805f1ea2-2a72-4abd-a100-fe217a03f97f" containerName="maas-api" containerID="cri-o://d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8" gracePeriod=30
Jul 14 19:28:32.083845 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.083812    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:28:32.312762 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.312699    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:28:32.412115 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.412080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805f1ea2-2a72-4abd-a100-fe217a03f97f-openshift-service-ca\") pod \"805f1ea2-2a72-4abd-a100-fe217a03f97f\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") "
Jul 14 19:28:32.412291 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.412129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805f1ea2-2a72-4abd-a100-fe217a03f97f-maas-api-tls\") pod \"805f1ea2-2a72-4abd-a100-fe217a03f97f\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") "
Jul 14 19:28:32.412291 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.412182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77wtv\" (UniqueName: \"kubernetes.io/projected/805f1ea2-2a72-4abd-a100-fe217a03f97f-kube-api-access-77wtv\") pod \"805f1ea2-2a72-4abd-a100-fe217a03f97f\" (UID: \"805f1ea2-2a72-4abd-a100-fe217a03f97f\") "
Jul 14 19:28:32.412494 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.412467    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805f1ea2-2a72-4abd-a100-fe217a03f97f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "805f1ea2-2a72-4abd-a100-fe217a03f97f" (UID: "805f1ea2-2a72-4abd-a100-fe217a03f97f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:28:32.414343 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.414313    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805f1ea2-2a72-4abd-a100-fe217a03f97f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "805f1ea2-2a72-4abd-a100-fe217a03f97f" (UID: "805f1ea2-2a72-4abd-a100-fe217a03f97f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:28:32.414506 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.414429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805f1ea2-2a72-4abd-a100-fe217a03f97f-kube-api-access-77wtv" (OuterVolumeSpecName: "kube-api-access-77wtv") pod "805f1ea2-2a72-4abd-a100-fe217a03f97f" (UID: "805f1ea2-2a72-4abd-a100-fe217a03f97f"). InnerVolumeSpecName "kube-api-access-77wtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:28:32.513738 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.513695    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/805f1ea2-2a72-4abd-a100-fe217a03f97f-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:32.513738 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.513729    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/805f1ea2-2a72-4abd-a100-fe217a03f97f-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:32.513738 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:32.513739    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wtv\" (UniqueName: \"kubernetes.io/projected/805f1ea2-2a72-4abd-a100-fe217a03f97f-kube-api-access-77wtv\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:33.048918 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.048878    2562 generic.go:356] "Generic (PLEG): container finished" podID="805f1ea2-2a72-4abd-a100-fe217a03f97f" containerID="d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8" exitCode=0
Jul 14 19:28:33.049078 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.048930    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"
Jul 14 19:28:33.049078 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.048939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q" event={"ID":"805f1ea2-2a72-4abd-a100-fe217a03f97f","Type":"ContainerDied","Data":"d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8"}
Jul 14 19:28:33.049078 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.048996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q" event={"ID":"805f1ea2-2a72-4abd-a100-fe217a03f97f","Type":"ContainerDied","Data":"dc6c1322b7e6be4baa4832812666bc81be17ac5827d508151d6b60301e9837ac"}
Jul 14 19:28:33.049078 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.049026    2562 scope.go:117] "RemoveContainer" containerID="d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8"
Jul 14 19:28:33.057829 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.057809    2562 scope.go:117] "RemoveContainer" containerID="d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8"
Jul 14 19:28:33.058085 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:28:33.058063    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8\": container with ID starting with d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8 not found: ID does not exist" containerID="d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8"
Jul 14 19:28:33.058140 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.058095    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8"} err="failed to get container status \"d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8\": rpc error: code = NotFound desc = could not find container \"d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8\": container with ID starting with d8d43cf8b178c94b6b2bcf2183700ec772308f1992ce5dce93f741651b36fed8 not found: ID does not exist"
Jul 14 19:28:33.067145 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.067114    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"]
Jul 14 19:28:33.072002 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:33.071979    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-618aef4e-74b9cb98df-zxv4q"]
Jul 14 19:28:34.621705 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:34.621675    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805f1ea2-2a72-4abd-a100-fe217a03f97f" path="/var/lib/kubelet/pods/805f1ea2-2a72-4abd-a100-fe217a03f97f/volumes"
Jul 14 19:28:36.649765 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:36.649726    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:28:41.633345 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:41.633254    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"]
Jul 14 19:28:41.678342 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:41.678303    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:28:56.654307 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:56.654263    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf" podUID="2e2c5d53-ef62-43c2-9b37-df465076aa9e" containerName="main" containerID="cri-o://a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae" gracePeriod=45
Jul 14 19:28:56.902443 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:56.902415    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:28:57.032295 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-model-cache\") pod \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") "
Jul 14 19:28:57.032295 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032306    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tmp-dir\") pod \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") "
Jul 14 19:28:57.032594 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032372    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tls-certs\") pod \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") "
Jul 14 19:28:57.032594 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q62mr\" (UniqueName: \"kubernetes.io/projected/2e2c5d53-ef62-43c2-9b37-df465076aa9e-kube-api-access-q62mr\") pod \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") "
Jul 14 19:28:57.032594 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032454    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-home\") pod \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") "
Jul 14 19:28:57.032594 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032487    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-dshm\") pod \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\" (UID: \"2e2c5d53-ef62-43c2-9b37-df465076aa9e\") "
Jul 14 19:28:57.032789 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032760    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-home" (OuterVolumeSpecName: "home") pod "2e2c5d53-ef62-43c2-9b37-df465076aa9e" (UID: "2e2c5d53-ef62-43c2-9b37-df465076aa9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:28:57.032921 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032894    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e2c5d53-ef62-43c2-9b37-df465076aa9e" (UID: "2e2c5d53-ef62-43c2-9b37-df465076aa9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:28:57.033047 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.032925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e2c5d53-ef62-43c2-9b37-df465076aa9e" (UID: "2e2c5d53-ef62-43c2-9b37-df465076aa9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:28:57.034927 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.034895    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-dshm" (OuterVolumeSpecName: "dshm") pod "2e2c5d53-ef62-43c2-9b37-df465076aa9e" (UID: "2e2c5d53-ef62-43c2-9b37-df465076aa9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:28:57.034927 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.034915    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e2c5d53-ef62-43c2-9b37-df465076aa9e" (UID: "2e2c5d53-ef62-43c2-9b37-df465076aa9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:28:57.035098 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.035011    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2c5d53-ef62-43c2-9b37-df465076aa9e-kube-api-access-q62mr" (OuterVolumeSpecName: "kube-api-access-q62mr") pod "2e2c5d53-ef62-43c2-9b37-df465076aa9e" (UID: "2e2c5d53-ef62-43c2-9b37-df465076aa9e"). InnerVolumeSpecName "kube-api-access-q62mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:28:57.133722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.133684    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:57.133722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.133710    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2c5d53-ef62-43c2-9b37-df465076aa9e-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:57.133722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.133727    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62mr\" (UniqueName: \"kubernetes.io/projected/2e2c5d53-ef62-43c2-9b37-df465076aa9e-kube-api-access-q62mr\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:57.133962 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.133737    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:57.133962 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.133747    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:57.133962 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.133754    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2c5d53-ef62-43c2-9b37-df465076aa9e-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:28:57.172496 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.172459    2562 generic.go:356] "Generic (PLEG): container finished" podID="2e2c5d53-ef62-43c2-9b37-df465076aa9e" containerID="a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae" exitCode=0
Jul 14 19:28:57.172676 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.172516    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"
Jul 14 19:28:57.172676 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.172518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf" event={"ID":"2e2c5d53-ef62-43c2-9b37-df465076aa9e","Type":"ContainerDied","Data":"a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae"}
Jul 14 19:28:57.172676 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.172626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf" event={"ID":"2e2c5d53-ef62-43c2-9b37-df465076aa9e","Type":"ContainerDied","Data":"e18b62676319c7e2a9f36ad439e5a1e77e84343282bb39dd13952b058dadced9"}
Jul 14 19:28:57.172676 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.172655    2562 scope.go:117] "RemoveContainer" containerID="a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae"
Jul 14 19:28:57.181872 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.181855    2562 scope.go:117] "RemoveContainer" containerID="a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae"
Jul 14 19:28:57.182154 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:28:57.182130    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae\": container with ID starting with a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae not found: ID does not exist" containerID="a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae"
Jul 14 19:28:57.182246 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.182161    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae"} err="failed to get container status \"a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae\": rpc error: code = NotFound desc = could not find container \"a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae\": container with ID starting with a216517761ab8fbbe9d19cd97e08ce859e7045ad50182709cf53a12e52f5a8ae not found: ID does not exist"
Jul 14 19:28:57.193975 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.193952    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"]
Jul 14 19:28:57.200283 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:57.200254    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-618aef4e/e2e-lifecycle-model-618aef4e-kserve-744889ddff-fgtkf"]
Jul 14 19:28:58.621730 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:28:58.621685    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2c5d53-ef62-43c2-9b37-df465076aa9e" path="/var/lib/kubelet/pods/2e2c5d53-ef62-43c2-9b37-df465076aa9e/volumes"
Jul 14 19:29:16.784879 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:16.784846    2562 scope.go:117] "RemoveContainer" containerID="cc5c531b3c698b1f192768e702d2773c53c547b810dcdc335b3af95d584084da"
Jul 14 19:29:17.864460 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:17.864397    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:19.037121 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:19.037083    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:25.030424 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:25.030360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:28.583462 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:28.583429    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:28.926490 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:28.926383    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:29.147262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:29.147228    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:30.175666 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:30.175629    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:30.273425 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:30.273367    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:33.764421 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:33.764364    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:33.864251 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:33.864197    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:38.902783 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:38.902748    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:38.975493 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:38.975452    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:39.580472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:39.580426    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:46.114285 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:46.114241    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:47.322669 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:47.322634    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:56.824655 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:56.824621    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:29:57.166427 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:29:57.166309    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:00.144159 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144122    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"]
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144465    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805f1ea2-2a72-4abd-a100-fe217a03f97f" containerName="maas-api"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144476    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="805f1ea2-2a72-4abd-a100-fe217a03f97f" containerName="maas-api"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144491    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c78ecefb-8dbb-4229-9778-b42b817a6be9" containerName="revoke-keys"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144497    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c78ecefb-8dbb-4229-9778-b42b817a6be9" containerName="revoke-keys"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144505    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2c5d53-ef62-43c2-9b37-df465076aa9e" containerName="main"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144511    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2c5d53-ef62-43c2-9b37-df465076aa9e" containerName="main"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144571    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="805f1ea2-2a72-4abd-a100-fe217a03f97f" containerName="maas-api"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144579    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2c5d53-ef62-43c2-9b37-df465076aa9e" containerName="main"
Jul 14 19:30:00.144601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.144586    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c78ecefb-8dbb-4229-9778-b42b817a6be9" containerName="revoke-keys"
Jul 14 19:30:00.147558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.147542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"
Jul 14 19:30:00.150347 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.150330    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:30:00.154530 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.154506    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"]
Jul 14 19:30:00.158951 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.158929    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"
Jul 14 19:30:00.294025 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.293994    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"]
Jul 14 19:30:00.295911 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:30:00.295885    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ee742_414b_4738_ae13_8762e3fc225d.slice/crio-cbf23c76968d73e196275fcc6bdbe0243349c48dde9363b95c5d445dd626883a WatchSource:0}: Error finding container cbf23c76968d73e196275fcc6bdbe0243349c48dde9363b95c5d445dd626883a: Status 404 returned error can't find the container with id cbf23c76968d73e196275fcc6bdbe0243349c48dde9363b95c5d445dd626883a
Jul 14 19:30:00.495681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:00.495647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs" event={"ID":"8b9ee742-414b-4738-ae13-8762e3fc225d","Type":"ContainerStarted","Data":"cbf23c76968d73e196275fcc6bdbe0243349c48dde9363b95c5d445dd626883a"}
Jul 14 19:30:01.503026 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:01.502930    2562 generic.go:356] "Generic (PLEG): container finished" podID="8b9ee742-414b-4738-ae13-8762e3fc225d" containerID="ca4305fedd682179f821ee0ba3471df773c99cceef728204ca7d8d2c9b9dc977" exitCode=0
Jul 14 19:30:01.503026 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:01.503018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs" event={"ID":"8b9ee742-414b-4738-ae13-8762e3fc225d","Type":"ContainerDied","Data":"ca4305fedd682179f821ee0ba3471df773c99cceef728204ca7d8d2c9b9dc977"}
Jul 14 19:30:02.635469 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:02.635439    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"
Jul 14 19:30:03.514321 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:03.514291    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs"
Jul 14 19:30:03.514321 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:03.514300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734290-4w7vs" event={"ID":"8b9ee742-414b-4738-ae13-8762e3fc225d","Type":"ContainerDied","Data":"cbf23c76968d73e196275fcc6bdbe0243349c48dde9363b95c5d445dd626883a"}
Jul 14 19:30:03.514610 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:03.514346    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf23c76968d73e196275fcc6bdbe0243349c48dde9363b95c5d445dd626883a"
Jul 14 19:30:03.870139 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:03.870054    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:16.810303 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:16.810224    2562 scope.go:117] "RemoveContainer" containerID="564df0730847e5708e00b7ebd4e0d3ad6cdfe3a21a6193827d2c5d676e7fc717"
Jul 14 19:30:18.134929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:18.134885    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:24.325676 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:24.325633    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:24.426651 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:24.426607    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:24.527349 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:24.527300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:24.913438 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:24.913378    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:25.046681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:25.046637    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:27.885551 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.885514    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:27.988204 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.988168    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:27.994938 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.994911    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"]
Jul 14 19:30:27.995244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.995233    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ee742-414b-4738-ae13-8762e3fc225d" containerName="cleanup"
Jul 14 19:30:27.995285 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.995246    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ee742-414b-4738-ae13-8762e3fc225d" containerName="cleanup"
Jul 14 19:30:27.995320 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.995314    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ee742-414b-4738-ae13-8762e3fc225d" containerName="cleanup"
Jul 14 19:30:27.998460 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:27.998443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.002080 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.002061    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-76d82b-serving-cert\""
Jul 14 19:30:28.002196 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.002130    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:30:28.003263 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.003242    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:30:28.003344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.003277    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:30:28.015538 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.015510    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"]
Jul 14 19:30:28.123318 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.123279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjgn\" (UniqueName: \"kubernetes.io/projected/d9005e7b-487c-4bd4-81ba-1be7d6398722-kube-api-access-czjgn\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.123537 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.123324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.123537 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.123356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9005e7b-487c-4bd4-81ba-1be7d6398722-openshift-service-ca\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.138021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.137933    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:28.224579 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.224545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9005e7b-487c-4bd4-81ba-1be7d6398722-openshift-service-ca\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.224769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.224643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjgn\" (UniqueName: \"kubernetes.io/projected/d9005e7b-487c-4bd4-81ba-1be7d6398722-kube-api-access-czjgn\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.224769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.224667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.224894 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:30:28.224785    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-serving-cert: secret "maas-api-e2e-inf-a-76d82b-serving-cert" not found
Jul 14 19:30:28.224894 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:30:28.224841    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls podName:d9005e7b-487c-4bd4-81ba-1be7d6398722 nodeName:}" failed. No retries permitted until 2026-07-14 19:30:28.724824996 +0000 UTC m=+2172.597383194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls") pod "maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" (UID: "d9005e7b-487c-4bd4-81ba-1be7d6398722") : secret "maas-api-e2e-inf-a-76d82b-serving-cert" not found
Jul 14 19:30:28.225252 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.225228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9005e7b-487c-4bd4-81ba-1be7d6398722-openshift-service-ca\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.238051 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.238026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjgn\" (UniqueName: \"kubernetes.io/projected/d9005e7b-487c-4bd4-81ba-1be7d6398722-kube-api-access-czjgn\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.484948 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.484910    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:28.672757 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.672690    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:28.729834 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.729786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.732322 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.732301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls\") pod \"maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:28.774089 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.773998    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:28.909302 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:28.909265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:29.056022 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:29.055994    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"]
Jul 14 19:30:29.057645 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:30:29.057612    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9005e7b_487c_4bd4_81ba_1be7d6398722.slice/crio-695f08186bbc97903dca10b9a5648b4464898da695fd6607a12e0920e5160cbc WatchSource:0}: Error finding container 695f08186bbc97903dca10b9a5648b4464898da695fd6607a12e0920e5160cbc: Status 404 returned error can't find the container with id 695f08186bbc97903dca10b9a5648b4464898da695fd6607a12e0920e5160cbc
Jul 14 19:30:29.642108 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:29.642067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" event={"ID":"d9005e7b-487c-4bd4-81ba-1be7d6398722","Type":"ContainerStarted","Data":"695f08186bbc97903dca10b9a5648b4464898da695fd6607a12e0920e5160cbc"}
Jul 14 19:30:30.647762 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:30.647713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" event={"ID":"d9005e7b-487c-4bd4-81ba-1be7d6398722","Type":"ContainerStarted","Data":"e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91"}
Jul 14 19:30:30.648144 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:30.647776    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:30.668666 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:30.668619    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" podStartSLOduration=3.064929234 podStartE2EDuration="3.668604449s" totalImagesPullingTime="603.675215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:30:30.667228411 +0000 UTC m=+2174.539786633" watchObservedRunningTime="2026-07-14 19:30:30.668604449 +0000 UTC m=+2174.541162668"
Jul 14 19:30:32.276056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:32.276027    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:32.619539 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:32.619445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:32.944807 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:32.944719    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:33.179523 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:33.179489    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:35.834043 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.834012    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:35.919056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.919024    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"]
Jul 14 19:30:35.923202 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.923178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:35.926059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.926041    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-feadd1-serving-cert\""
Jul 14 19:30:35.932556 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.932531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"]
Jul 14 19:30:35.935470 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.935445    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:35.994009 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.993969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:35.994218 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.994019    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea8c4bf-900c-4986-b898-54bef8d02cdf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:35.994218 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:35.994149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8c7d\" (UniqueName: \"kubernetes.io/projected/8ea8c4bf-900c-4986-b898-54bef8d02cdf-kube-api-access-k8c7d\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.029829 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.029791    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:36.095669 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.095580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.095669 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.095630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea8c4bf-900c-4986-b898-54bef8d02cdf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.095862 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.095698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8c7d\" (UniqueName: \"kubernetes.io/projected/8ea8c4bf-900c-4986-b898-54bef8d02cdf-kube-api-access-k8c7d\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.095862 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:30:36.095745    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-serving-cert: secret "maas-api-e2e-inf-b-feadd1-serving-cert" not found
Jul 14 19:30:36.095862 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:30:36.095811    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls podName:8ea8c4bf-900c-4986-b898-54bef8d02cdf nodeName:}" failed. No retries permitted until 2026-07-14 19:30:36.595794908 +0000 UTC m=+2180.468353108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls") pod "maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" (UID: "8ea8c4bf-900c-4986-b898-54bef8d02cdf") : secret "maas-api-e2e-inf-b-feadd1-serving-cert" not found
Jul 14 19:30:36.096389 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.096368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea8c4bf-900c-4986-b898-54bef8d02cdf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.108925 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.108893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8c7d\" (UniqueName: \"kubernetes.io/projected/8ea8c4bf-900c-4986-b898-54bef8d02cdf-kube-api-access-k8c7d\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.459143 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.459044    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:36.598015 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.597977    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:36.600094 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.600073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.602624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.602592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls\") pod \"maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.658949 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.658922    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:30:36.706740 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.706701    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:36.834825 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.834787    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:36.991272 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:36.991247    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"]
Jul 14 19:30:36.993760 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:30:36.993733    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea8c4bf_900c_4986_b898_54bef8d02cdf.slice/crio-83bebf3870459740a3ae6277c52fcfde584e66fa19bcc52f5cbad6e0507ab516 WatchSource:0}: Error finding container 83bebf3870459740a3ae6277c52fcfde584e66fa19bcc52f5cbad6e0507ab516: Status 404 returned error can't find the container with id 83bebf3870459740a3ae6277c52fcfde584e66fa19bcc52f5cbad6e0507ab516
Jul 14 19:30:37.680693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:37.680592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" event={"ID":"8ea8c4bf-900c-4986-b898-54bef8d02cdf","Type":"ContainerStarted","Data":"7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e"}
Jul 14 19:30:37.680693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:37.680664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" event={"ID":"8ea8c4bf-900c-4986-b898-54bef8d02cdf","Type":"ContainerStarted","Data":"83bebf3870459740a3ae6277c52fcfde584e66fa19bcc52f5cbad6e0507ab516"}
Jul 14 19:30:37.680885 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:37.680731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:37.707895 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:37.707846    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" podStartSLOduration=2.338818819 podStartE2EDuration="2.70782945s" totalImagesPullingTime="369.010631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:30:37.706210265 +0000 UTC m=+2181.578768498" watchObservedRunningTime="2026-07-14 19:30:37.70782945 +0000 UTC m=+2181.580387671"
Jul 14 19:30:40.361053 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:40.361022    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:43.691588 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:43.691560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:30:46.685315 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.685276    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"]
Jul 14 19:30:46.688935 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.688911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.695052 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.695025    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-76d82b\"/\"default-dockercfg-pz8nb\""
Jul 14 19:30:46.695219 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.695056    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-76d82b\"/\"kube-root-ca.crt\""
Jul 14 19:30:46.695219 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.695106    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-76d82b\"/\"test-model-76d82b-kserve-self-signed-certs\""
Jul 14 19:30:46.695219 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.695056    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-76d82b\"/\"openshift-service-ca.crt\""
Jul 14 19:30:46.700418 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.700381    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"]
Jul 14 19:30:46.766953 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.766913    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:46.799916 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.799883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tls-certs\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.800091 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.799957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-dshm\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.800091 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.800007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tmp-dir\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.800091 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.800022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-home\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.800091 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.800066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-model-cache\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.800273 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.800095    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cxs\" (UniqueName: \"kubernetes.io/projected/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-kube-api-access-b5cxs\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.900846 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.900810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-dshm\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.900993 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.900876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tmp-dir\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.900993 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.900904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-home\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.900993 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.900926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-model-cache\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.900993 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.900948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cxs\" (UniqueName: \"kubernetes.io/projected/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-kube-api-access-b5cxs\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.901173 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.900996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tls-certs\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.901393 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.901358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-home\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.901558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.901465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-model-cache\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.901558 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.901497    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tmp-dir\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.903354 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.903330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-dshm\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.903478 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.903460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tls-certs\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.911572 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.911546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cxs\" (UniqueName: \"kubernetes.io/projected/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-kube-api-access-b5cxs\") pod \"test-model-76d82b-kserve-6f4c49d4f9-jdj7g\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") " pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:46.998479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:46.998443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:47.141731 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:47.141701    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"]
Jul 14 19:30:47.143523 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:30:47.143496    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6cb2c51_dcfe_44c8_92af_bf8a99dbb970.slice/crio-97f209f0cbe5418c7ff1a0cf7fec65d18925c42b47ff0f0ed51ea8ba8c1e71a2 WatchSource:0}: Error finding container 97f209f0cbe5418c7ff1a0cf7fec65d18925c42b47ff0f0ed51ea8ba8c1e71a2: Status 404 returned error can't find the container with id 97f209f0cbe5418c7ff1a0cf7fec65d18925c42b47ff0f0ed51ea8ba8c1e71a2
Jul 14 19:30:47.732319 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:47.732286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g" event={"ID":"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970","Type":"ContainerStarted","Data":"28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925"}
Jul 14 19:30:47.732724 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:47.732341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g" event={"ID":"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970","Type":"ContainerStarted","Data":"97f209f0cbe5418c7ff1a0cf7fec65d18925c42b47ff0f0ed51ea8ba8c1e71a2"}
Jul 14 19:30:47.756459 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:47.756382    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g" podStartSLOduration=1.7563659569999999 podStartE2EDuration="1.756365957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:30:47.754454351 +0000 UTC m=+2191.627012571" watchObservedRunningTime="2026-07-14 19:30:47.756365957 +0000 UTC m=+2191.628924175"
Jul 14 19:30:49.777281 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:49.777239    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:49.880033 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:49.879973    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:56.998820 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:56.998781    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:56.999295 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:56.998846    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:57.012603 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:57.012575    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:57.705396 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:57.705360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:30:57.793420 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:57.793372    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:30:57.809039 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:30:57.809003    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:16.670672 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.670620    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"]
Jul 14 19:31:16.674607 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.674581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.677387 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.677362    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-feadd1\"/\"kube-root-ca.crt\""
Jul 14 19:31:16.677567 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.677420    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-feadd1\"/\"openshift-service-ca.crt\""
Jul 14 19:31:16.678481 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.678464    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-feadd1\"/\"test-model-feadd1-kserve-self-signed-certs\""
Jul 14 19:31:16.678605 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.678508    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-feadd1\"/\"default-dockercfg-5qtfx\""
Jul 14 19:31:16.683982 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.683961    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"]
Jul 14 19:31:16.704124 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.704082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:16.777703 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.777671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-dshm\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.777703 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.777710    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-home\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.777922 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.777755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbvr\" (UniqueName: \"kubernetes.io/projected/481c5b52-9cbb-4639-b57b-c056ce06f665-kube-api-access-vjbvr\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.777922 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.777777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-tmp-dir\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.777922 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.777824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481c5b52-9cbb-4639-b57b-c056ce06f665-tls-certs\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.777922 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.777872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-model-cache\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-dshm\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-home\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879352 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbvr\" (UniqueName: \"kubernetes.io/projected/481c5b52-9cbb-4639-b57b-c056ce06f665-kube-api-access-vjbvr\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879352 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-tmp-dir\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879352 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481c5b52-9cbb-4639-b57b-c056ce06f665-tls-certs\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879352 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-model-cache\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879617 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-home\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879660 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-tmp-dir\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.879755 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.879733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-model-cache\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.881578 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.881555    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-dshm\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.881870 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.881853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481c5b52-9cbb-4639-b57b-c056ce06f665-tls-certs\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.888237 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.888211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbvr\" (UniqueName: \"kubernetes.io/projected/481c5b52-9cbb-4639-b57b-c056ce06f665-kube-api-access-vjbvr\") pod \"test-model-feadd1-kserve-67f748684c-qndpt\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") " pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:16.985896 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:16.985852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:17.130381 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:17.130294    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"]
Jul 14 19:31:17.132873 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:31:17.132832    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod481c5b52_9cbb_4639_b57b_c056ce06f665.slice/crio-c5d3d8c73c03c6a88d83f9b516be091358f9a2f7fcc49a607b849ce114845cbe WatchSource:0}: Error finding container c5d3d8c73c03c6a88d83f9b516be091358f9a2f7fcc49a607b849ce114845cbe: Status 404 returned error can't find the container with id c5d3d8c73c03c6a88d83f9b516be091358f9a2f7fcc49a607b849ce114845cbe
Jul 14 19:31:17.880100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:17.880060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt" event={"ID":"481c5b52-9cbb-4639-b57b-c056ce06f665","Type":"ContainerStarted","Data":"ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2"}
Jul 14 19:31:17.880563 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:17.880109    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt" event={"ID":"481c5b52-9cbb-4639-b57b-c056ce06f665","Type":"ContainerStarted","Data":"c5d3d8c73c03c6a88d83f9b516be091358f9a2f7fcc49a607b849ce114845cbe"}
Jul 14 19:31:17.899818 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:17.899766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt" podStartSLOduration=1.899750875 podStartE2EDuration="1.899750875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:31:17.898110191 +0000 UTC m=+2221.770668413" watchObservedRunningTime="2026-07-14 19:31:17.899750875 +0000 UTC m=+2221.772309095"
Jul 14 19:31:26.986031 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:26.985987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:26.986647 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:26.986051    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:26.998829 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:26.998791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:27.945197 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:27.945160    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:39.082063 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:39.081981    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"]
Jul 14 19:31:39.091117 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:39.091072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:40.010071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:40.010039    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"]
Jul 14 19:31:40.031631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:40.031591    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:40.776340 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:40.776300    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:40.877647 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:40.877611    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:50.754297 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:50.754254    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"]
Jul 14 19:31:50.754842 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:50.754541    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" podUID="d9005e7b-487c-4bd4-81ba-1be7d6398722" containerName="maas-api" containerID="cri-o://e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91" gracePeriod=30
Jul 14 19:31:50.787568 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:50.787533    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:51.003886 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.003863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:31:51.047910 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.047821    2562 generic.go:356] "Generic (PLEG): container finished" podID="d9005e7b-487c-4bd4-81ba-1be7d6398722" containerID="e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91" exitCode=0
Jul 14 19:31:51.047910 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.047866    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"
Jul 14 19:31:51.048142 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.047906    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" event={"ID":"d9005e7b-487c-4bd4-81ba-1be7d6398722","Type":"ContainerDied","Data":"e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91"}
Jul 14 19:31:51.048142 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.047962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77" event={"ID":"d9005e7b-487c-4bd4-81ba-1be7d6398722","Type":"ContainerDied","Data":"695f08186bbc97903dca10b9a5648b4464898da695fd6607a12e0920e5160cbc"}
Jul 14 19:31:51.048142 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.047992    2562 scope.go:117] "RemoveContainer" containerID="e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91"
Jul 14 19:31:51.056660 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.056644    2562 scope.go:117] "RemoveContainer" containerID="e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91"
Jul 14 19:31:51.056920 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:31:51.056903    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91\": container with ID starting with e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91 not found: ID does not exist" containerID="e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91"
Jul 14 19:31:51.056959 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.056931    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91"} err="failed to get container status \"e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91\": rpc error: code = NotFound desc = could not find container \"e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91\": container with ID starting with e2b8b997c96e1b7e50acb6852ed9d80498b974fe3dbb7b3ad47a5ec0ef284a91 not found: ID does not exist"
Jul 14 19:31:51.097301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.097267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls\") pod \"d9005e7b-487c-4bd4-81ba-1be7d6398722\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") "
Jul 14 19:31:51.097477 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.097355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9005e7b-487c-4bd4-81ba-1be7d6398722-openshift-service-ca\") pod \"d9005e7b-487c-4bd4-81ba-1be7d6398722\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") "
Jul 14 19:31:51.097477 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.097394    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czjgn\" (UniqueName: \"kubernetes.io/projected/d9005e7b-487c-4bd4-81ba-1be7d6398722-kube-api-access-czjgn\") pod \"d9005e7b-487c-4bd4-81ba-1be7d6398722\" (UID: \"d9005e7b-487c-4bd4-81ba-1be7d6398722\") "
Jul 14 19:31:51.097706 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.097681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9005e7b-487c-4bd4-81ba-1be7d6398722-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9005e7b-487c-4bd4-81ba-1be7d6398722" (UID: "d9005e7b-487c-4bd4-81ba-1be7d6398722"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:31:51.099490 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.099467    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9005e7b-487c-4bd4-81ba-1be7d6398722" (UID: "d9005e7b-487c-4bd4-81ba-1be7d6398722"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:31:51.099587 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.099577    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9005e7b-487c-4bd4-81ba-1be7d6398722-kube-api-access-czjgn" (OuterVolumeSpecName: "kube-api-access-czjgn") pod "d9005e7b-487c-4bd4-81ba-1be7d6398722" (UID: "d9005e7b-487c-4bd4-81ba-1be7d6398722"). InnerVolumeSpecName "kube-api-access-czjgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:31:51.198299 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.198244    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9005e7b-487c-4bd4-81ba-1be7d6398722-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:51.198299 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.198293    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czjgn\" (UniqueName: \"kubernetes.io/projected/d9005e7b-487c-4bd4-81ba-1be7d6398722-kube-api-access-czjgn\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:51.198299 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.198307    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9005e7b-487c-4bd4-81ba-1be7d6398722-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:51.370740 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.370704    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"]
Jul 14 19:31:51.375251 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:51.375218    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-76d82b-5bdbdcd556-hsm77"]
Jul 14 19:31:52.621140 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:52.621102    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9005e7b-487c-4bd4-81ba-1be7d6398722" path="/var/lib/kubelet/pods/d9005e7b-487c-4bd4-81ba-1be7d6398722/volumes"
Jul 14 19:31:54.102912 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.102853    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g" podUID="b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" containerName="main" containerID="cri-o://28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925" gracePeriod=45
Jul 14 19:31:54.353451 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.353379    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:31:54.428344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428306    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-model-cache\") pod \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") "
Jul 14 19:31:54.428344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tls-certs\") pod \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") "
Jul 14 19:31:54.428641 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428374    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tmp-dir\") pod \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") "
Jul 14 19:31:54.428641 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428441    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5cxs\" (UniqueName: \"kubernetes.io/projected/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-kube-api-access-b5cxs\") pod \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") "
Jul 14 19:31:54.428641 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-dshm\") pod \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") "
Jul 14 19:31:54.428641 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428502    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-home\") pod \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\" (UID: \"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970\") "
Jul 14 19:31:54.428836 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" (UID: "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:54.428836 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" (UID: "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:54.428944 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-home" (OuterVolumeSpecName: "home") pod "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" (UID: "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:54.428944 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428885    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:54.428944 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.428929    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:54.430844 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.430823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" (UID: "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:31:54.430947 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.430843    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-dshm" (OuterVolumeSpecName: "dshm") pod "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" (UID: "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:54.430947 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.430889    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-kube-api-access-b5cxs" (OuterVolumeSpecName: "kube-api-access-b5cxs") pod "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" (UID: "b6cb2c51-dcfe-44c8-92af-bf8a99dbb970"). InnerVolumeSpecName "kube-api-access-b5cxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:31:54.529603 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.529568    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:54.529603 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.529595    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5cxs\" (UniqueName: \"kubernetes.io/projected/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-kube-api-access-b5cxs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:54.529603 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.529605    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:54.529603 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:54.529614    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:55.031061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.031010    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt" podUID="481c5b52-9cbb-4639-b57b-c056ce06f665" containerName="main" containerID="cri-o://ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2" gracePeriod=45
Jul 14 19:31:55.069307 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.069274    2562 generic.go:356] "Generic (PLEG): container finished" podID="b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" containerID="28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925" exitCode=0
Jul 14 19:31:55.069509 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.069331    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"
Jul 14 19:31:55.069509 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.069335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g" event={"ID":"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970","Type":"ContainerDied","Data":"28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925"}
Jul 14 19:31:55.069509 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.069378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g" event={"ID":"b6cb2c51-dcfe-44c8-92af-bf8a99dbb970","Type":"ContainerDied","Data":"97f209f0cbe5418c7ff1a0cf7fec65d18925c42b47ff0f0ed51ea8ba8c1e71a2"}
Jul 14 19:31:55.069509 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.069414    2562 scope.go:117] "RemoveContainer" containerID="28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925"
Jul 14 19:31:55.112369 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.112338    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"]
Jul 14 19:31:55.115573 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.115547    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-76d82b/test-model-76d82b-kserve-6f4c49d4f9-jdj7g"]
Jul 14 19:31:55.142442 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.142148    2562 scope.go:117] "RemoveContainer" containerID="28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925"
Jul 14 19:31:55.142534 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:31:55.142513    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925\": container with ID starting with 28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925 not found: ID does not exist" containerID="28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925"
Jul 14 19:31:55.142597 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.142543    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925"} err="failed to get container status \"28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925\": rpc error: code = NotFound desc = could not find container \"28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925\": container with ID starting with 28aa74301e3464562c20bb0f5809ceb8f74e5043dc120368b5b6b9ab355c9925 not found: ID does not exist"
Jul 14 19:31:55.277952 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.277915    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:31:55.290627 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.290568    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:55.336450 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-dshm\") pod \"481c5b52-9cbb-4639-b57b-c056ce06f665\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") "
Jul 14 19:31:55.336645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336460    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjbvr\" (UniqueName: \"kubernetes.io/projected/481c5b52-9cbb-4639-b57b-c056ce06f665-kube-api-access-vjbvr\") pod \"481c5b52-9cbb-4639-b57b-c056ce06f665\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") "
Jul 14 19:31:55.336645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336478    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-tmp-dir\") pod \"481c5b52-9cbb-4639-b57b-c056ce06f665\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") "
Jul 14 19:31:55.336645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336525    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-home\") pod \"481c5b52-9cbb-4639-b57b-c056ce06f665\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") "
Jul 14 19:31:55.336645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-model-cache\") pod \"481c5b52-9cbb-4639-b57b-c056ce06f665\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") "
Jul 14 19:31:55.336645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481c5b52-9cbb-4639-b57b-c056ce06f665-tls-certs\") pod \"481c5b52-9cbb-4639-b57b-c056ce06f665\" (UID: \"481c5b52-9cbb-4639-b57b-c056ce06f665\") "
Jul 14 19:31:55.336915 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336812    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "481c5b52-9cbb-4639-b57b-c056ce06f665" (UID: "481c5b52-9cbb-4639-b57b-c056ce06f665"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:55.336915 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336830    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-home" (OuterVolumeSpecName: "home") pod "481c5b52-9cbb-4639-b57b-c056ce06f665" (UID: "481c5b52-9cbb-4639-b57b-c056ce06f665"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:55.336915 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-model-cache" (OuterVolumeSpecName: "model-cache") pod "481c5b52-9cbb-4639-b57b-c056ce06f665" (UID: "481c5b52-9cbb-4639-b57b-c056ce06f665"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:55.337061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.336984    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:55.337061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.337003    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:55.337061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.337039    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:55.338985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.338955    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481c5b52-9cbb-4639-b57b-c056ce06f665-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "481c5b52-9cbb-4639-b57b-c056ce06f665" (UID: "481c5b52-9cbb-4639-b57b-c056ce06f665"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:31:55.338985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.338970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-dshm" (OuterVolumeSpecName: "dshm") pod "481c5b52-9cbb-4639-b57b-c056ce06f665" (UID: "481c5b52-9cbb-4639-b57b-c056ce06f665"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:31:55.339166 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.339007    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481c5b52-9cbb-4639-b57b-c056ce06f665-kube-api-access-vjbvr" (OuterVolumeSpecName: "kube-api-access-vjbvr") pod "481c5b52-9cbb-4639-b57b-c056ce06f665" (UID: "481c5b52-9cbb-4639-b57b-c056ce06f665"). InnerVolumeSpecName "kube-api-access-vjbvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:31:55.438087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.438055    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/481c5b52-9cbb-4639-b57b-c056ce06f665-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:55.438087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.438080    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/481c5b52-9cbb-4639-b57b-c056ce06f665-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:55.438087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:55.438091    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbvr\" (UniqueName: \"kubernetes.io/projected/481c5b52-9cbb-4639-b57b-c056ce06f665-kube-api-access-vjbvr\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:31:56.075722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.075689    2562 generic.go:356] "Generic (PLEG): container finished" podID="481c5b52-9cbb-4639-b57b-c056ce06f665" containerID="ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2" exitCode=0
Jul 14 19:31:56.075929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.075742    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"
Jul 14 19:31:56.075929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.075769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt" event={"ID":"481c5b52-9cbb-4639-b57b-c056ce06f665","Type":"ContainerDied","Data":"ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2"}
Jul 14 19:31:56.075929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.075813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt" event={"ID":"481c5b52-9cbb-4639-b57b-c056ce06f665","Type":"ContainerDied","Data":"c5d3d8c73c03c6a88d83f9b516be091358f9a2f7fcc49a607b849ce114845cbe"}
Jul 14 19:31:56.075929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.075849    2562 scope.go:117] "RemoveContainer" containerID="ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2"
Jul 14 19:31:56.084999 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.084978    2562 scope.go:117] "RemoveContainer" containerID="ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2"
Jul 14 19:31:56.085255 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:31:56.085238    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2\": container with ID starting with ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2 not found: ID does not exist" containerID="ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2"
Jul 14 19:31:56.085316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.085262    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2"} err="failed to get container status \"ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2\": rpc error: code = NotFound desc = could not find container \"ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2\": container with ID starting with ab72c6b4ba165a78d2d4d5105181081e17b5cf48844736bc623822a99b07fdd2 not found: ID does not exist"
Jul 14 19:31:56.105439 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.105388    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"]
Jul 14 19:31:56.114562 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.114536    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-feadd1/test-model-feadd1-kserve-67f748684c-qndpt"]
Jul 14 19:31:56.621036 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.621004    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481c5b52-9cbb-4639-b57b-c056ce06f665" path="/var/lib/kubelet/pods/481c5b52-9cbb-4639-b57b-c056ce06f665/volumes"
Jul 14 19:31:56.621397 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:31:56.621384    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" path="/var/lib/kubelet/pods/b6cb2c51-dcfe-44c8-92af-bf8a99dbb970/volumes"
Jul 14 19:32:01.712233 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:01.712194    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:01.811220 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:01.811180    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:06.083293 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.083252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:06.366137 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.366052    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:06.476157 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476123    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"]
Jul 14 19:32:06.476504 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476490    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" containerName="main"
Jul 14 19:32:06.476555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476506    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" containerName="main"
Jul 14 19:32:06.476555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476518    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481c5b52-9cbb-4639-b57b-c056ce06f665" containerName="main"
Jul 14 19:32:06.476555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476525    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="481c5b52-9cbb-4639-b57b-c056ce06f665" containerName="main"
Jul 14 19:32:06.476555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476534    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9005e7b-487c-4bd4-81ba-1be7d6398722" containerName="maas-api"
Jul 14 19:32:06.476555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476539    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9005e7b-487c-4bd4-81ba-1be7d6398722" containerName="maas-api"
Jul 14 19:32:06.476707 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476588    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cb2c51-dcfe-44c8-92af-bf8a99dbb970" containerName="main"
Jul 14 19:32:06.476707 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476598    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9005e7b-487c-4bd4-81ba-1be7d6398722" containerName="maas-api"
Jul 14 19:32:06.476707 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.476606    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="481c5b52-9cbb-4639-b57b-c056ce06f665" containerName="main"
Jul 14 19:32:06.481145 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.481124    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:06.484329 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.484307    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:32:06.488988 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.488824    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"]
Jul 14 19:32:06.536768 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.536729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2ae5f3-c73d-4e55-8807-82dad39c30a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k\" (UID: \"cb2ae5f3-c73d-4e55-8807-82dad39c30a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:06.637945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.637841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2ae5f3-c73d-4e55-8807-82dad39c30a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k\" (UID: \"cb2ae5f3-c73d-4e55-8807-82dad39c30a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:06.638555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.638535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2ae5f3-c73d-4e55-8807-82dad39c30a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k\" (UID: \"cb2ae5f3-c73d-4e55-8807-82dad39c30a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:06.792975 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.792942    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:06.934512 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:06.934481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"]
Jul 14 19:32:06.936457 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:32:06.936433    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2ae5f3_c73d_4e55_8807_82dad39c30a0.slice/crio-979d6ecfbaee0f92a97c7c2ae0246926ce9138b9a62e782e2e88ddd6a5e891ee WatchSource:0}: Error finding container 979d6ecfbaee0f92a97c7c2ae0246926ce9138b9a62e782e2e88ddd6a5e891ee: Status 404 returned error can't find the container with id 979d6ecfbaee0f92a97c7c2ae0246926ce9138b9a62e782e2e88ddd6a5e891ee
Jul 14 19:32:07.135968 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:07.135933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k" event={"ID":"cb2ae5f3-c73d-4e55-8807-82dad39c30a0","Type":"ContainerStarted","Data":"979d6ecfbaee0f92a97c7c2ae0246926ce9138b9a62e782e2e88ddd6a5e891ee"}
Jul 14 19:32:07.223804 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:07.223776    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:07.418305 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:07.418272    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:08.142043 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:08.142007    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb2ae5f3-c73d-4e55-8807-82dad39c30a0" containerID="10b696db182ddc557c07d686c702c7e4114e43c7d75975a1657a720488fcf2a7" exitCode=0
Jul 14 19:32:08.142433 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:08.142052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k" event={"ID":"cb2ae5f3-c73d-4e55-8807-82dad39c30a0","Type":"ContainerDied","Data":"10b696db182ddc557c07d686c702c7e4114e43c7d75975a1657a720488fcf2a7"}
Jul 14 19:32:09.298552 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:09.298528    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:09.369782 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:09.369746    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2ae5f3-c73d-4e55-8807-82dad39c30a0-maas-api-service-ca\") pod \"cb2ae5f3-c73d-4e55-8807-82dad39c30a0\" (UID: \"cb2ae5f3-c73d-4e55-8807-82dad39c30a0\") "
Jul 14 19:32:09.370159 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:09.370133    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2ae5f3-c73d-4e55-8807-82dad39c30a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb2ae5f3-c73d-4e55-8807-82dad39c30a0" (UID: "cb2ae5f3-c73d-4e55-8807-82dad39c30a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:32:09.470872 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:09.470842    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2ae5f3-c73d-4e55-8807-82dad39c30a0-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:32:10.153346 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:10.153315    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k"
Jul 14 19:32:10.153346 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:10.153329    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k" event={"ID":"cb2ae5f3-c73d-4e55-8807-82dad39c30a0","Type":"ContainerDied","Data":"979d6ecfbaee0f92a97c7c2ae0246926ce9138b9a62e782e2e88ddd6a5e891ee"}
Jul 14 19:32:10.153581 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:10.153372    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979d6ecfbaee0f92a97c7c2ae0246926ce9138b9a62e782e2e88ddd6a5e891ee"
Jul 14 19:32:11.057679 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:11.057628    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:11.157919 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:11.157887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:17.235118 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.235053    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:17.364531 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.364487    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:17.567539 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.567431    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"]
Jul 14 19:32:17.567707 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.567645    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" podUID="8ea8c4bf-900c-4986-b898-54bef8d02cdf" containerName="maas-api" containerID="cri-o://7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e" gracePeriod=30
Jul 14 19:32:17.599645 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.599594    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:17.822642 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.822573    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:32:17.948142 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.948105    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls\") pod \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") "
Jul 14 19:32:17.948339 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.948152    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea8c4bf-900c-4986-b898-54bef8d02cdf-openshift-service-ca\") pod \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") "
Jul 14 19:32:17.948339 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.948182    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8c7d\" (UniqueName: \"kubernetes.io/projected/8ea8c4bf-900c-4986-b898-54bef8d02cdf-kube-api-access-k8c7d\") pod \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\" (UID: \"8ea8c4bf-900c-4986-b898-54bef8d02cdf\") "
Jul 14 19:32:17.948598 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.948573    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea8c4bf-900c-4986-b898-54bef8d02cdf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ea8c4bf-900c-4986-b898-54bef8d02cdf" (UID: "8ea8c4bf-900c-4986-b898-54bef8d02cdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:32:17.950458 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.950434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ea8c4bf-900c-4986-b898-54bef8d02cdf" (UID: "8ea8c4bf-900c-4986-b898-54bef8d02cdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:32:17.950561 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:17.950545    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea8c4bf-900c-4986-b898-54bef8d02cdf-kube-api-access-k8c7d" (OuterVolumeSpecName: "kube-api-access-k8c7d") pod "8ea8c4bf-900c-4986-b898-54bef8d02cdf" (UID: "8ea8c4bf-900c-4986-b898-54bef8d02cdf"). InnerVolumeSpecName "kube-api-access-k8c7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:32:18.049701 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.049664    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ea8c4bf-900c-4986-b898-54bef8d02cdf-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:32:18.049701 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.049694    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea8c4bf-900c-4986-b898-54bef8d02cdf-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:32:18.049701 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.049705    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8c7d\" (UniqueName: \"kubernetes.io/projected/8ea8c4bf-900c-4986-b898-54bef8d02cdf-kube-api-access-k8c7d\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:32:18.194083 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.193985    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ea8c4bf-900c-4986-b898-54bef8d02cdf" containerID="7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e" exitCode=0
Jul 14 19:32:18.194083 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.194053    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"
Jul 14 19:32:18.194277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.194076    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" event={"ID":"8ea8c4bf-900c-4986-b898-54bef8d02cdf","Type":"ContainerDied","Data":"7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e"}
Jul 14 19:32:18.194277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.194163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg" event={"ID":"8ea8c4bf-900c-4986-b898-54bef8d02cdf","Type":"ContainerDied","Data":"83bebf3870459740a3ae6277c52fcfde584e66fa19bcc52f5cbad6e0507ab516"}
Jul 14 19:32:18.194277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.194193    2562 scope.go:117] "RemoveContainer" containerID="7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e"
Jul 14 19:32:18.203254 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.203231    2562 scope.go:117] "RemoveContainer" containerID="7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e"
Jul 14 19:32:18.203568 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:32:18.203544    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e\": container with ID starting with 7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e not found: ID does not exist" containerID="7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e"
Jul 14 19:32:18.203673 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.203574    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e"} err="failed to get container status \"7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e\": rpc error: code = NotFound desc = could not find container \"7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e\": container with ID starting with 7ba4d53549cc2b3f9ea438cf31de5a7ba52031ed291e01f3a11b2703b76d5f1e not found: ID does not exist"
Jul 14 19:32:18.218783 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.218747    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"]
Jul 14 19:32:18.225194 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.225169    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-feadd1-58db4f6945-fj6gg"]
Jul 14 19:32:18.621273 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:18.621237    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea8c4bf-900c-4986-b898-54bef8d02cdf" path="/var/lib/kubelet/pods/8ea8c4bf-900c-4986-b898-54bef8d02cdf/volumes"
Jul 14 19:32:21.597168 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:21.597138    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:28.745345 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:28.745298    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:28.918289 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:28.918252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:32.432061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:32.432018    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:35.631146 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:35.631110    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:35.744002 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:35.743969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:36.571039 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.570999    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:36.711945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.711912    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:36.728423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728370    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"]
Jul 14 19:32:36.728764 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728751    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea8c4bf-900c-4986-b898-54bef8d02cdf" containerName="maas-api"
Jul 14 19:32:36.728810 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728766    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea8c4bf-900c-4986-b898-54bef8d02cdf" containerName="maas-api"
Jul 14 19:32:36.728810 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728777    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2ae5f3-c73d-4e55-8807-82dad39c30a0" containerName="revoke-keys"
Jul 14 19:32:36.728810 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728783    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2ae5f3-c73d-4e55-8807-82dad39c30a0" containerName="revoke-keys"
Jul 14 19:32:36.728908 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728835    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2ae5f3-c73d-4e55-8807-82dad39c30a0" containerName="revoke-keys"
Jul 14 19:32:36.728908 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.728847    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea8c4bf-900c-4986-b898-54bef8d02cdf" containerName="maas-api"
Jul 14 19:32:36.731899 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.731884    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.734847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.734824    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-de8e7b-serving-cert\""
Jul 14 19:32:36.734966 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.734918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:32:36.735966 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.735950    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:32:36.736087 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.735969    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:32:36.759541 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.759508    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"]
Jul 14 19:32:36.804918 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.804879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7bea2-ab75-4847-9086-5fb3f56c4117-openshift-service-ca\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.805083 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.804959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkz2\" (UniqueName: \"kubernetes.io/projected/b0a7bea2-ab75-4847-9086-5fb3f56c4117-kube-api-access-vbkz2\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.805132 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.805088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.850143 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.850062    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:36.905785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.905750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkz2\" (UniqueName: \"kubernetes.io/projected/b0a7bea2-ab75-4847-9086-5fb3f56c4117-kube-api-access-vbkz2\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.905961 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.905842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.905961 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.905876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7bea2-ab75-4847-9086-5fb3f56c4117-openshift-service-ca\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.906057 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:32:36.905985    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-serving-cert: secret "maas-api-e2e-api-a-de8e7b-serving-cert" not found
Jul 14 19:32:36.906101 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:32:36.906065    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls podName:b0a7bea2-ab75-4847-9086-5fb3f56c4117 nodeName:}" failed. No retries permitted until 2026-07-14 19:32:37.406043708 +0000 UTC m=+2301.278601907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls") pod "maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" (UID: "b0a7bea2-ab75-4847-9086-5fb3f56c4117") : secret "maas-api-e2e-api-a-de8e7b-serving-cert" not found
Jul 14 19:32:36.906590 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.906573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7bea2-ab75-4847-9086-5fb3f56c4117-openshift-service-ca\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:36.928366 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:36.928330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkz2\" (UniqueName: \"kubernetes.io/projected/b0a7bea2-ab75-4847-9086-5fb3f56c4117-kube-api-access-vbkz2\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:37.101127 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.101036    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:37.410209 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.410124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:37.412607 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.412582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls\") pod \"maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:37.591099 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.591066    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:37.641547 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.641510    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:37.698921 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.698883    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:37.796260 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.796197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"]
Jul 14 19:32:37.798665 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:32:37.798639    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a7bea2_ab75_4847_9086_5fb3f56c4117.slice/crio-b2e6defb4d950315baedc9ab365576838a2f27143e739b751b40406500cfd214 WatchSource:0}: Error finding container b2e6defb4d950315baedc9ab365576838a2f27143e739b751b40406500cfd214: Status 404 returned error can't find the container with id b2e6defb4d950315baedc9ab365576838a2f27143e739b751b40406500cfd214
Jul 14 19:32:37.800331 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:37.800313    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 19:32:38.304377 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:38.304344    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" event={"ID":"b0a7bea2-ab75-4847-9086-5fb3f56c4117","Type":"ContainerStarted","Data":"b2e6defb4d950315baedc9ab365576838a2f27143e739b751b40406500cfd214"}
Jul 14 19:32:39.310638 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:39.310596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" event={"ID":"b0a7bea2-ab75-4847-9086-5fb3f56c4117","Type":"ContainerStarted","Data":"bd4104640fa925ee3c64d537fcbbeeeb72ac5bdb86f6b6f1a751459b31a5d686"}
Jul 14 19:32:39.311020 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:39.310792    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:39.338101 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:39.338051    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" podStartSLOduration=2.8508203500000002 podStartE2EDuration="3.338035069s" totalImagesPullingTime="487.214719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:32:39.335568545 +0000 UTC m=+2303.208126776" watchObservedRunningTime="2026-07-14 19:32:39.338035069 +0000 UTC m=+2303.210593288"
Jul 14 19:32:41.092691 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:41.092642    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:41.680803 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:41.680768    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:42.180917 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:42.180884    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:42.503838 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:42.503789    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:45.321825 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:45.321799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:32:45.860840 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:45.860798    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:45.984102 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:45.984073    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"]
Jul 14 19:32:45.987267 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:45.987251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:45.991071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:45.991046    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8a10a8-serving-cert\""
Jul 14 19:32:46.011734 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.011703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"]
Jul 14 19:32:46.070658 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.070615    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:46.076680 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.076655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fsxk\" (UniqueName: \"kubernetes.io/projected/1d069f29-8dac-4137-b5a7-32ac4efe9a25-kube-api-access-6fsxk\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.076785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.076688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d069f29-8dac-4137-b5a7-32ac4efe9a25-openshift-service-ca\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.076785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.076713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.178089 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.178001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fsxk\" (UniqueName: \"kubernetes.io/projected/1d069f29-8dac-4137-b5a7-32ac4efe9a25-kube-api-access-6fsxk\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.178089 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.178046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d069f29-8dac-4137-b5a7-32ac4efe9a25-openshift-service-ca\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.178268 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.178167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.178319 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:32:46.178302    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-serving-cert: secret "maas-api-e2e-api-b-8a10a8-serving-cert" not found
Jul 14 19:32:46.178378 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:32:46.178364    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls podName:1d069f29-8dac-4137-b5a7-32ac4efe9a25 nodeName:}" failed. No retries permitted until 2026-07-14 19:32:46.678347848 +0000 UTC m=+2310.550906045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls") pod "maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" (UID: "1d069f29-8dac-4137-b5a7-32ac4efe9a25") : secret "maas-api-e2e-api-b-8a10a8-serving-cert" not found
Jul 14 19:32:46.178651 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.178636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d069f29-8dac-4137-b5a7-32ac4efe9a25-openshift-service-ca\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.202376 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.202345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fsxk\" (UniqueName: \"kubernetes.io/projected/1d069f29-8dac-4137-b5a7-32ac4efe9a25-kube-api-access-6fsxk\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.281139 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.281099    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:46.463441 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.463387    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:46.684351 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.684300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.686915 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.686893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls\") pod \"maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:46.897706 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:46.897606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:47.006611 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:47.006562    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:47.090352 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:47.090311    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"]
Jul 14 19:32:47.092887 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:32:47.092854    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d069f29_8dac_4137_b5a7_32ac4efe9a25.slice/crio-2f9d2ea614e209aebda7a0e85898436087c1c4e85b2ae7402257a11b920d160c WatchSource:0}: Error finding container 2f9d2ea614e209aebda7a0e85898436087c1c4e85b2ae7402257a11b920d160c: Status 404 returned error can't find the container with id 2f9d2ea614e209aebda7a0e85898436087c1c4e85b2ae7402257a11b920d160c
Jul 14 19:32:47.206948 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:47.206861    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:47.349899 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:47.349854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" event={"ID":"1d069f29-8dac-4137-b5a7-32ac4efe9a25","Type":"ContainerStarted","Data":"2f9d2ea614e209aebda7a0e85898436087c1c4e85b2ae7402257a11b920d160c"}
Jul 14 19:32:48.356784 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:48.356753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" event={"ID":"1d069f29-8dac-4137-b5a7-32ac4efe9a25","Type":"ContainerStarted","Data":"1073164714b80a0b8beab4c8d42274ef792b85cca8b238b221eb960e73a490a6"}
Jul 14 19:32:48.357192 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:48.356823    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:50.305694 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:50.305665    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:54.366674 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:54.366645    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:32:54.391803 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:54.391749    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" podStartSLOduration=8.812478339 podStartE2EDuration="9.391735235s" totalImagesPullingTime="579.256896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:32:48.405852518 +0000 UTC m=+2312.278410740" watchObservedRunningTime="2026-07-14 19:32:54.391735235 +0000 UTC m=+2318.264293456"
Jul 14 19:32:58.465542 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.465499    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"]
Jul 14 19:32:58.469025 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.469003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:32:58.472262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.472241    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:32:58.492090 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.492054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"]
Jul 14 19:32:58.588049 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.588016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c627736-fe42-4a52-ab27-fb9ad03e31a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5\" (UID: \"2c627736-fe42-4a52-ab27-fb9ad03e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:32:58.689260 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.689219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c627736-fe42-4a52-ab27-fb9ad03e31a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5\" (UID: \"2c627736-fe42-4a52-ab27-fb9ad03e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:32:58.689955 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.689930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c627736-fe42-4a52-ab27-fb9ad03e31a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5\" (UID: \"2c627736-fe42-4a52-ab27-fb9ad03e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:32:58.780142 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.780057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:32:58.930652 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.930628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"]
Jul 14 19:32:58.970323 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:58.970295    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:59.094049 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:59.093961    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:32:59.410458 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:59.410373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5" event={"ID":"2c627736-fe42-4a52-ab27-fb9ad03e31a3","Type":"ContainerStarted","Data":"85e4f112ca17a092c2d0045b7e3f62aa0eecbd97b282acb832a30bbdaa71f19f"}
Jul 14 19:32:59.410458 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:32:59.410452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5" event={"ID":"2c627736-fe42-4a52-ab27-fb9ad03e31a3","Type":"ContainerStarted","Data":"a0707802b3a11f1e8520e662051ce4e822b0c17cc99d16f7d8b45ec46a06e258"}
Jul 14 19:33:00.416472 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:00.416437    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c627736-fe42-4a52-ab27-fb9ad03e31a3" containerID="85e4f112ca17a092c2d0045b7e3f62aa0eecbd97b282acb832a30bbdaa71f19f" exitCode=0
Jul 14 19:33:00.416889 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:00.416531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5" event={"ID":"2c627736-fe42-4a52-ab27-fb9ad03e31a3","Type":"ContainerDied","Data":"85e4f112ca17a092c2d0045b7e3f62aa0eecbd97b282acb832a30bbdaa71f19f"}
Jul 14 19:33:01.551981 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:01.551954    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:33:01.616693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:01.616658    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c627736-fe42-4a52-ab27-fb9ad03e31a3-maas-api-service-ca\") pod \"2c627736-fe42-4a52-ab27-fb9ad03e31a3\" (UID: \"2c627736-fe42-4a52-ab27-fb9ad03e31a3\") "
Jul 14 19:33:01.617049 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:01.617024    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c627736-fe42-4a52-ab27-fb9ad03e31a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c627736-fe42-4a52-ab27-fb9ad03e31a3" (UID: "2c627736-fe42-4a52-ab27-fb9ad03e31a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:33:01.717796 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:01.717770    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c627736-fe42-4a52-ab27-fb9ad03e31a3-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:02.428783 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:02.428753    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5"
Jul 14 19:33:02.428972 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:02.428755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5" event={"ID":"2c627736-fe42-4a52-ab27-fb9ad03e31a3","Type":"ContainerDied","Data":"a0707802b3a11f1e8520e662051ce4e822b0c17cc99d16f7d8b45ec46a06e258"}
Jul 14 19:33:02.428972 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:02.428874    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0707802b3a11f1e8520e662051ce4e822b0c17cc99d16f7d8b45ec46a06e258"
Jul 14 19:33:08.131745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:08.131650    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:08.252189 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:08.252149    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:09.320362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.320324    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"]
Jul 14 19:33:09.320837 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.320589    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" podUID="b0a7bea2-ab75-4847-9086-5fb3f56c4117" containerName="maas-api" containerID="cri-o://bd4104640fa925ee3c64d537fcbbeeeb72ac5bdb86f6b6f1a751459b31a5d686" gracePeriod=30
Jul 14 19:33:09.407579 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.407531    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:09.463499 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.463459    2562 generic.go:356] "Generic (PLEG): container finished" podID="b0a7bea2-ab75-4847-9086-5fb3f56c4117" containerID="bd4104640fa925ee3c64d537fcbbeeeb72ac5bdb86f6b6f1a751459b31a5d686" exitCode=0
Jul 14 19:33:09.463782 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.463538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" event={"ID":"b0a7bea2-ab75-4847-9086-5fb3f56c4117","Type":"ContainerDied","Data":"bd4104640fa925ee3c64d537fcbbeeeb72ac5bdb86f6b6f1a751459b31a5d686"}
Jul 14 19:33:09.564585 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.564554    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:33:09.685706 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.685617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls\") pod \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") "
Jul 14 19:33:09.685706 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.685659    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbkz2\" (UniqueName: \"kubernetes.io/projected/b0a7bea2-ab75-4847-9086-5fb3f56c4117-kube-api-access-vbkz2\") pod \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") "
Jul 14 19:33:09.685706 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.685682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7bea2-ab75-4847-9086-5fb3f56c4117-openshift-service-ca\") pod \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\" (UID: \"b0a7bea2-ab75-4847-9086-5fb3f56c4117\") "
Jul 14 19:33:09.686109 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.686070    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a7bea2-ab75-4847-9086-5fb3f56c4117-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0a7bea2-ab75-4847-9086-5fb3f56c4117" (UID: "b0a7bea2-ab75-4847-9086-5fb3f56c4117"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:33:09.687925 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.687901    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a7bea2-ab75-4847-9086-5fb3f56c4117-kube-api-access-vbkz2" (OuterVolumeSpecName: "kube-api-access-vbkz2") pod "b0a7bea2-ab75-4847-9086-5fb3f56c4117" (UID: "b0a7bea2-ab75-4847-9086-5fb3f56c4117"). InnerVolumeSpecName "kube-api-access-vbkz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:33:09.688004 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.687926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0a7bea2-ab75-4847-9086-5fb3f56c4117" (UID: "b0a7bea2-ab75-4847-9086-5fb3f56c4117"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:33:09.786328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.786297    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0a7bea2-ab75-4847-9086-5fb3f56c4117-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:09.786328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.786322    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0a7bea2-ab75-4847-9086-5fb3f56c4117-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:09.786328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:09.786331    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbkz2\" (UniqueName: \"kubernetes.io/projected/b0a7bea2-ab75-4847-9086-5fb3f56c4117-kube-api-access-vbkz2\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:10.470794 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:10.470763    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"
Jul 14 19:33:10.470794 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:10.470772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk" event={"ID":"b0a7bea2-ab75-4847-9086-5fb3f56c4117","Type":"ContainerDied","Data":"b2e6defb4d950315baedc9ab365576838a2f27143e739b751b40406500cfd214"}
Jul 14 19:33:10.471297 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:10.470838    2562 scope.go:117] "RemoveContainer" containerID="bd4104640fa925ee3c64d537fcbbeeeb72ac5bdb86f6b6f1a751459b31a5d686"
Jul 14 19:33:10.496896 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:10.496863    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"]
Jul 14 19:33:10.501939 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:10.501912    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de8e7b-7946cb658f-fdtjk"]
Jul 14 19:33:10.620950 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:10.620920    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0a7bea2-ab75-4847-9086-5fb3f56c4117" path="/var/lib/kubelet/pods/b0a7bea2-ab75-4847-9086-5fb3f56c4117/volumes"
Jul 14 19:33:13.652002 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:13.651969    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:20.533132 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:20.533092    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:20.624182 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:20.624153    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:24.369109 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.369072    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:24.575892 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.575850    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:24.702870 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.702787    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"]
Jul 14 19:33:24.703143 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.703131    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0a7bea2-ab75-4847-9086-5fb3f56c4117" containerName="maas-api"
Jul 14 19:33:24.703188 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.703145    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0a7bea2-ab75-4847-9086-5fb3f56c4117" containerName="maas-api"
Jul 14 19:33:24.703188 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.703165    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c627736-fe42-4a52-ab27-fb9ad03e31a3" containerName="revoke-keys"
Jul 14 19:33:24.703188 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.703171    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c627736-fe42-4a52-ab27-fb9ad03e31a3" containerName="revoke-keys"
Jul 14 19:33:24.703280 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.703227    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c627736-fe42-4a52-ab27-fb9ad03e31a3" containerName="revoke-keys"
Jul 14 19:33:24.703280 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.703234    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0a7bea2-ab75-4847-9086-5fb3f56c4117" containerName="maas-api"
Jul 14 19:33:24.706287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.706266    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:24.709083 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.709064    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:33:24.718096 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.718066    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"]
Jul 14 19:33:24.827905 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.827866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339b04d3-bb53-4daf-9748-4fca5bda59fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4\" (UID: \"339b04d3-bb53-4daf-9748-4fca5bda59fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:24.928513 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.928476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339b04d3-bb53-4daf-9748-4fca5bda59fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4\" (UID: \"339b04d3-bb53-4daf-9748-4fca5bda59fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:24.929165 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:24.929136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339b04d3-bb53-4daf-9748-4fca5bda59fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4\" (UID: \"339b04d3-bb53-4daf-9748-4fca5bda59fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:25.016899 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:25.016853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:25.160191 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:25.160167    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"]
Jul 14 19:33:25.162177 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:33:25.162134    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339b04d3_bb53_4daf_9748_4fca5bda59fe.slice/crio-d1bd69ce15d52a2e02b991c4e27cb5f90746b85a77118c5dd3bf4e5f2289c622 WatchSource:0}: Error finding container d1bd69ce15d52a2e02b991c4e27cb5f90746b85a77118c5dd3bf4e5f2289c622: Status 404 returned error can't find the container with id d1bd69ce15d52a2e02b991c4e27cb5f90746b85a77118c5dd3bf4e5f2289c622
Jul 14 19:33:25.545985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:25.545951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4" event={"ID":"339b04d3-bb53-4daf-9748-4fca5bda59fe","Type":"ContainerStarted","Data":"7862dd88b9f7ecf259f578515cb86fb53d4b9a8bc61f684b02405adc16f03add"}
Jul 14 19:33:25.546442 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:25.545999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4" event={"ID":"339b04d3-bb53-4daf-9748-4fca5bda59fe","Type":"ContainerStarted","Data":"d1bd69ce15d52a2e02b991c4e27cb5f90746b85a77118c5dd3bf4e5f2289c622"}
Jul 14 19:33:26.552583 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:26.552544    2562 generic.go:356] "Generic (PLEG): container finished" podID="339b04d3-bb53-4daf-9748-4fca5bda59fe" containerID="7862dd88b9f7ecf259f578515cb86fb53d4b9a8bc61f684b02405adc16f03add" exitCode=0
Jul 14 19:33:26.553037 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:26.552639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4" event={"ID":"339b04d3-bb53-4daf-9748-4fca5bda59fe","Type":"ContainerDied","Data":"7862dd88b9f7ecf259f578515cb86fb53d4b9a8bc61f684b02405adc16f03add"}
Jul 14 19:33:27.683113 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:27.683082    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:27.748684 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:27.748655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339b04d3-bb53-4daf-9748-4fca5bda59fe-maas-api-service-ca\") pod \"339b04d3-bb53-4daf-9748-4fca5bda59fe\" (UID: \"339b04d3-bb53-4daf-9748-4fca5bda59fe\") "
Jul 14 19:33:27.749005 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:27.748982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339b04d3-bb53-4daf-9748-4fca5bda59fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "339b04d3-bb53-4daf-9748-4fca5bda59fe" (UID: "339b04d3-bb53-4daf-9748-4fca5bda59fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:33:27.849913 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:27.849821    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/339b04d3-bb53-4daf-9748-4fca5bda59fe-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:28.563210 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:28.563171    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4" event={"ID":"339b04d3-bb53-4daf-9748-4fca5bda59fe","Type":"ContainerDied","Data":"d1bd69ce15d52a2e02b991c4e27cb5f90746b85a77118c5dd3bf4e5f2289c622"}
Jul 14 19:33:28.563457 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:28.563252    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1bd69ce15d52a2e02b991c4e27cb5f90746b85a77118c5dd3bf4e5f2289c622"
Jul 14 19:33:28.563457 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:28.563184    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4"
Jul 14 19:33:35.379045 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.379007    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"]
Jul 14 19:33:35.379614 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.379235    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" podUID="1d069f29-8dac-4137-b5a7-32ac4efe9a25" containerName="maas-api" containerID="cri-o://1073164714b80a0b8beab4c8d42274ef792b85cca8b238b221eb960e73a490a6" gracePeriod=30
Jul 14 19:33:35.421715 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.421680    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:35.598755 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.598717    2562 generic.go:356] "Generic (PLEG): container finished" podID="1d069f29-8dac-4137-b5a7-32ac4efe9a25" containerID="1073164714b80a0b8beab4c8d42274ef792b85cca8b238b221eb960e73a490a6" exitCode=0
Jul 14 19:33:35.598755 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.598757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" event={"ID":"1d069f29-8dac-4137-b5a7-32ac4efe9a25","Type":"ContainerDied","Data":"1073164714b80a0b8beab4c8d42274ef792b85cca8b238b221eb960e73a490a6"}
Jul 14 19:33:35.624916 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.624890    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:33:35.719150 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.719121    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fsxk\" (UniqueName: \"kubernetes.io/projected/1d069f29-8dac-4137-b5a7-32ac4efe9a25-kube-api-access-6fsxk\") pod \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") "
Jul 14 19:33:35.719356 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.719165    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls\") pod \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") "
Jul 14 19:33:35.719356 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.719189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d069f29-8dac-4137-b5a7-32ac4efe9a25-openshift-service-ca\") pod \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\" (UID: \"1d069f29-8dac-4137-b5a7-32ac4efe9a25\") "
Jul 14 19:33:35.719592 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.719567    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d069f29-8dac-4137-b5a7-32ac4efe9a25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d069f29-8dac-4137-b5a7-32ac4efe9a25" (UID: "1d069f29-8dac-4137-b5a7-32ac4efe9a25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:33:35.721380 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.721348    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d069f29-8dac-4137-b5a7-32ac4efe9a25-kube-api-access-6fsxk" (OuterVolumeSpecName: "kube-api-access-6fsxk") pod "1d069f29-8dac-4137-b5a7-32ac4efe9a25" (UID: "1d069f29-8dac-4137-b5a7-32ac4efe9a25"). InnerVolumeSpecName "kube-api-access-6fsxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:33:35.721380 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.721366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d069f29-8dac-4137-b5a7-32ac4efe9a25" (UID: "1d069f29-8dac-4137-b5a7-32ac4efe9a25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:33:35.820062 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.820017    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fsxk\" (UniqueName: \"kubernetes.io/projected/1d069f29-8dac-4137-b5a7-32ac4efe9a25-kube-api-access-6fsxk\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:35.820062 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.820055    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d069f29-8dac-4137-b5a7-32ac4efe9a25-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:35.820062 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:35.820067    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d069f29-8dac-4137-b5a7-32ac4efe9a25-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:33:36.605501 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:36.605464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv" event={"ID":"1d069f29-8dac-4137-b5a7-32ac4efe9a25","Type":"ContainerDied","Data":"2f9d2ea614e209aebda7a0e85898436087c1c4e85b2ae7402257a11b920d160c"}
Jul 14 19:33:36.605960 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:36.605530    2562 scope.go:117] "RemoveContainer" containerID="1073164714b80a0b8beab4c8d42274ef792b85cca8b238b221eb960e73a490a6"
Jul 14 19:33:36.605960 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:36.605481    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"
Jul 14 19:33:36.629724 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:36.629693    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"]
Jul 14 19:33:36.638040 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:36.638011    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8a10a8-65898d94bd-tsrgv"]
Jul 14 19:33:38.621192 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:38.621160    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d069f29-8dac-4137-b5a7-32ac4efe9a25" path="/var/lib/kubelet/pods/1d069f29-8dac-4137-b5a7-32ac4efe9a25/volumes"
Jul 14 19:33:39.822293 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:39.822257    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:45.773004 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:45.772965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:46.023162 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:46.023067    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:46.109042 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:46.109006    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:50.527576 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:50.527540    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:50.753582 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:50.753547    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:51.166922 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:51.166864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:51.408171 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:51.408132    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:52.308816 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:52.308778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:53.961623 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:53.961590    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:54.062577 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.062514    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"]
Jul 14 19:33:54.062960 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.062941    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d069f29-8dac-4137-b5a7-32ac4efe9a25" containerName="maas-api"
Jul 14 19:33:54.063059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.062964    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d069f29-8dac-4137-b5a7-32ac4efe9a25" containerName="maas-api"
Jul 14 19:33:54.063059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.062988    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="339b04d3-bb53-4daf-9748-4fca5bda59fe" containerName="revoke-keys"
Jul 14 19:33:54.063059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.062996    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="339b04d3-bb53-4daf-9748-4fca5bda59fe" containerName="revoke-keys"
Jul 14 19:33:54.063207 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.063093    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="339b04d3-bb53-4daf-9748-4fca5bda59fe" containerName="revoke-keys"
Jul 14 19:33:54.063207 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.063106    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d069f29-8dac-4137-b5a7-32ac4efe9a25" containerName="maas-api"
Jul 14 19:33:54.066224 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.066203    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.069810 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.069778    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:54.070593 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.070575    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-755a0b-serving-cert\""
Jul 14 19:33:54.070715 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.070603    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 19:33:54.072144 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.072125    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-82sfz\""
Jul 14 19:33:54.072343 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.072331    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 19:33:54.086694 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.086666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"]
Jul 14 19:33:54.187026 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.186988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.187226 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.187042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c402efd7-2d02-45e4-a94c-e65e09da4c39-openshift-service-ca\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.187226 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.187116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpcn\" (UniqueName: \"kubernetes.io/projected/c402efd7-2d02-45e4-a94c-e65e09da4c39-kube-api-access-4rpcn\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.222813 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.222779    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:54.288426 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.288368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpcn\" (UniqueName: \"kubernetes.io/projected/c402efd7-2d02-45e4-a94c-e65e09da4c39-kube-api-access-4rpcn\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.288601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.288488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.288601 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.288523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c402efd7-2d02-45e4-a94c-e65e09da4c39-openshift-service-ca\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.288673 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:33:54.288621    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-serving-cert: secret "maas-api-e2e-shared-a-755a0b-serving-cert" not found
Jul 14 19:33:54.288753 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:33:54.288693    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls podName:c402efd7-2d02-45e4-a94c-e65e09da4c39 nodeName:}" failed. No retries permitted until 2026-07-14 19:33:54.788677761 +0000 UTC m=+2378.661235959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls") pod "maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" (UID: "c402efd7-2d02-45e4-a94c-e65e09da4c39") : secret "maas-api-e2e-shared-a-755a0b-serving-cert" not found
Jul 14 19:33:54.289098 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.289081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c402efd7-2d02-45e4-a94c-e65e09da4c39-openshift-service-ca\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.305478 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.305454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpcn\" (UniqueName: \"kubernetes.io/projected/c402efd7-2d02-45e4-a94c-e65e09da4c39-kube-api-access-4rpcn\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.367348 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.367312    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:54.794375 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.794345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.796291 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.796264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:54.797140 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.797124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls\") pod \"maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:54.896418 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.896374    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:54.977070 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:54.977037    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:55.119423 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:55.119370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"]
Jul 14 19:33:55.121893 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:33:55.121864    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc402efd7_2d02_45e4_a94c_e65e09da4c39.slice/crio-637f41edbde363dc0136f62fab642b7b5b40f369ef9e91ab2566740ab5b02eee WatchSource:0}: Error finding container 637f41edbde363dc0136f62fab642b7b5b40f369ef9e91ab2566740ab5b02eee: Status 404 returned error can't find the container with id 637f41edbde363dc0136f62fab642b7b5b40f369ef9e91ab2566740ab5b02eee
Jul 14 19:33:55.699429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:55.699373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" event={"ID":"c402efd7-2d02-45e4-a94c-e65e09da4c39","Type":"ContainerStarted","Data":"c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c"}
Jul 14 19:33:55.699609 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:55.699442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" event={"ID":"c402efd7-2d02-45e4-a94c-e65e09da4c39","Type":"ContainerStarted","Data":"637f41edbde363dc0136f62fab642b7b5b40f369ef9e91ab2566740ab5b02eee"}
Jul 14 19:33:55.699609 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:55.699479    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:33:55.719611 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:55.719513    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" podStartSLOduration=1.376817157 podStartE2EDuration="1.71949545s" totalImagesPullingTime="342.678293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:33:55.717985893 +0000 UTC m=+2379.590544115" watchObservedRunningTime="2026-07-14 19:33:55.71949545 +0000 UTC m=+2379.592053669"
Jul 14 19:33:58.398442 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:58.398375    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:33:58.743014 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:33:58.742980    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:01.712109 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:01.712078    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:34:02.222994 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.222965    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:02.299308 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.299271    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"]
Jul 14 19:34:02.302847 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.302830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.305844 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.305824    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-7c2fbe-serving-cert\""
Jul 14 19:34:02.331722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.331691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"]
Jul 14 19:34:02.383156 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.383105    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:02.467815 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.467770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1441875-274d-4cf6-a565-49e9df408627-maas-api-tls\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.467989 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.467830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrcm6\" (UniqueName: \"kubernetes.io/projected/e1441875-274d-4cf6-a565-49e9df408627-kube-api-access-hrcm6\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.467989 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.467857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1441875-274d-4cf6-a565-49e9df408627-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.479190 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.479101    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:02.569059 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.569025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrcm6\" (UniqueName: \"kubernetes.io/projected/e1441875-274d-4cf6-a565-49e9df408627-kube-api-access-hrcm6\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.569228 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.569065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1441875-274d-4cf6-a565-49e9df408627-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.569228 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.569144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1441875-274d-4cf6-a565-49e9df408627-maas-api-tls\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.569785 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.569765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1441875-274d-4cf6-a565-49e9df408627-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.571693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.571675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1441875-274d-4cf6-a565-49e9df408627-maas-api-tls\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.579640 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.579615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrcm6\" (UniqueName: \"kubernetes.io/projected/e1441875-274d-4cf6-a565-49e9df408627-kube-api-access-hrcm6\") pod \"maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.612515 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.612480    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:02.762328 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.762298    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"]
Jul 14 19:34:02.763580 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:34:02.763552    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1441875_274d_4cf6_a565_49e9df408627.slice/crio-f6ec718711e823bfd7f84e84dd72cb9c80ff4c8a048bf59844d86a6a18b73be8 WatchSource:0}: Error finding container f6ec718711e823bfd7f84e84dd72cb9c80ff4c8a048bf59844d86a6a18b73be8: Status 404 returned error can't find the container with id f6ec718711e823bfd7f84e84dd72cb9c80ff4c8a048bf59844d86a6a18b73be8
Jul 14 19:34:02.898287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.898252    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:02.998062 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:02.998031    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:03.741640 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:03.741604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb" event={"ID":"e1441875-274d-4cf6-a565-49e9df408627","Type":"ContainerStarted","Data":"5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb"}
Jul 14 19:34:03.741844 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:03.741662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb" event={"ID":"e1441875-274d-4cf6-a565-49e9df408627","Type":"ContainerStarted","Data":"f6ec718711e823bfd7f84e84dd72cb9c80ff4c8a048bf59844d86a6a18b73be8"}
Jul 14 19:34:03.741844 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:03.741704    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:03.761497 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:03.761449    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb" podStartSLOduration=1.421121324 podStartE2EDuration="1.761433541s" totalImagesPullingTime="340.312217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:34:03.759869674 +0000 UTC m=+2387.632427909" watchObservedRunningTime="2026-07-14 19:34:03.761433541 +0000 UTC m=+2387.633991787"
Jul 14 19:34:09.497283 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:09.497203    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:09.752745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:09.752670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:34:15.770985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:15.770939    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:15.855989 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:15.855943    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:24.982135 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:24.982093    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:25.083069 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.083029    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:25.175084 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.175052    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"]
Jul 14 19:34:25.178548 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.178528    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.182262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.182234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"openshift-service-ca.crt\""
Jul 14 19:34:25.182262 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.182263    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"kube-root-ca.crt\""
Jul 14 19:34:25.182499 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.182296    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"auth-test-model-755a0b-kserve-self-signed-certs\""
Jul 14 19:34:25.182499 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.182239    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"default-dockercfg-8j764\""
Jul 14 19:34:25.191231 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.191210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"]
Jul 14 19:34:25.195108 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.195082    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:25.277035 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.276927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-model-cache\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.277035 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.276981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-dshm\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.277035 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.277033    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-home\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.277277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.277057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355312b8-77b6-4626-82ee-6880185141b1-tls-certs\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.277277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.277136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22tc\" (UniqueName: \"kubernetes.io/projected/355312b8-77b6-4626-82ee-6880185141b1-kube-api-access-x22tc\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.277277 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.277239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-tmp-dir\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-tmp-dir\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-model-cache\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378569 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-dshm\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378569 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-home\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378569 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355312b8-77b6-4626-82ee-6880185141b1-tls-certs\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378569 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22tc\" (UniqueName: \"kubernetes.io/projected/355312b8-77b6-4626-82ee-6880185141b1-kube-api-access-x22tc\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378795 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378772    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-tmp-dir\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378886 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-home\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.378886 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.378885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-model-cache\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.380825 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.380800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-dshm\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.381160 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.381143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355312b8-77b6-4626-82ee-6880185141b1-tls-certs\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.387350 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.387324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22tc\" (UniqueName: \"kubernetes.io/projected/355312b8-77b6-4626-82ee-6880185141b1-kube-api-access-x22tc\") pod \"auth-test-model-755a0b-kserve-6775688c95-8l2sn\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") " pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.487777 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.487724    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:25.629818 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.629793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"]
Jul 14 19:34:25.631631 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:34:25.631596    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod355312b8_77b6_4626_82ee_6880185141b1.slice/crio-c59f1171afdd5fb9d179c13ca2126396c892e20dcaaec2b877ed896a92cf9087 WatchSource:0}: Error finding container c59f1171afdd5fb9d179c13ca2126396c892e20dcaaec2b877ed896a92cf9087: Status 404 returned error can't find the container with id c59f1171afdd5fb9d179c13ca2126396c892e20dcaaec2b877ed896a92cf9087
Jul 14 19:34:25.853807 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.853710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn" event={"ID":"355312b8-77b6-4626-82ee-6880185141b1","Type":"ContainerStarted","Data":"119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6"}
Jul 14 19:34:25.853807 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.853758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn" event={"ID":"355312b8-77b6-4626-82ee-6880185141b1","Type":"ContainerStarted","Data":"c59f1171afdd5fb9d179c13ca2126396c892e20dcaaec2b877ed896a92cf9087"}
Jul 14 19:34:25.874276 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:25.874218    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn" podStartSLOduration=0.874201624 podStartE2EDuration="874.201624ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:34:25.872455787 +0000 UTC m=+2409.745014007" watchObservedRunningTime="2026-07-14 19:34:25.874201624 +0000 UTC m=+2409.746759897"
Jul 14 19:34:35.488413 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:35.488371    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:35.488901 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:35.488455    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:35.501005 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:35.500975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:35.914615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:35.914529    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:34:38.860963 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:38.860879    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:38.930109 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:38.930064    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:44.231431 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:44.231360    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:56.576185 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.576149    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"]
Jul 14 19:34:56.579631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.579604    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.583306 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.583282    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"default-dockercfg-xphld\""
Jul 14 19:34:56.583461 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.583283    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"auth-test-model-7c2fbe-kserve-self-signed-certs\""
Jul 14 19:34:56.583461 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.583284    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"openshift-service-ca.crt\""
Jul 14 19:34:56.583461 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.583290    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"kube-root-ca.crt\""
Jul 14 19:34:56.590720 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.590697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"]
Jul 14 19:34:56.604544 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.604511    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:34:56.669490 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.669452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-model-cache\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.669490 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.669488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkqn\" (UniqueName: \"kubernetes.io/projected/66c1bd37-b765-4a1a-8182-98f3774e51fb-kube-api-access-gdkqn\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.669709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.669510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-tmp-dir\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.669709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.669587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-dshm\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.669709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.669654    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1bd37-b765-4a1a-8182-98f3774e51fb-tls-certs\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.669709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.669682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-home\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770466 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-home\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770466 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-model-cache\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkqn\" (UniqueName: \"kubernetes.io/projected/66c1bd37-b765-4a1a-8182-98f3774e51fb-kube-api-access-gdkqn\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-tmp-dir\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-dshm\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1bd37-b765-4a1a-8182-98f3774e51fb-tls-certs\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770915 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-model-cache\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770950 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-home\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.770985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.770954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-tmp-dir\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.773053 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.773034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-dshm\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.773173 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.773156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1bd37-b765-4a1a-8182-98f3774e51fb-tls-certs\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.779431 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.779386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkqn\" (UniqueName: \"kubernetes.io/projected/66c1bd37-b765-4a1a-8182-98f3774e51fb-kube-api-access-gdkqn\") pod \"auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") " pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:56.889302 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:56.889202    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:34:57.036386 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:57.036356    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"]
Jul 14 19:34:57.038378 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:34:57.038345    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66c1bd37_b765_4a1a_8182_98f3774e51fb.slice/crio-7cc7de06b499591f6f2af09c79f886229fd50c3c42536e98a72e80ef9af65028 WatchSource:0}: Error finding container 7cc7de06b499591f6f2af09c79f886229fd50c3c42536e98a72e80ef9af65028: Status 404 returned error can't find the container with id 7cc7de06b499591f6f2af09c79f886229fd50c3c42536e98a72e80ef9af65028
Jul 14 19:34:58.015037 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:58.014993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg" event={"ID":"66c1bd37-b765-4a1a-8182-98f3774e51fb","Type":"ContainerStarted","Data":"d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6"}
Jul 14 19:34:58.015450 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:58.015046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg" event={"ID":"66c1bd37-b765-4a1a-8182-98f3774e51fb","Type":"ContainerStarted","Data":"7cc7de06b499591f6f2af09c79f886229fd50c3c42536e98a72e80ef9af65028"}
Jul 14 19:34:58.045875 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:34:58.045819    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg" podStartSLOduration=2.045802328 podStartE2EDuration="2.045802328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:34:58.043224333 +0000 UTC m=+2441.915782554" watchObservedRunningTime="2026-07-14 19:34:58.045802328 +0000 UTC m=+2441.918360547"
Jul 14 19:35:04.548672 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:35:04.548631    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:35:06.889681 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:35:06.889625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:35:06.890259 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:35:06.889716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:35:06.902098 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:35:06.902072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:35:07.074636 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:35:07.074603    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:35:15.703025 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:35:15.702983    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:36:51.831482 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:36:51.831441    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"]
Jul 14 19:36:51.937883 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:36:51.937845    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:36:52.688167 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:36:52.688127    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"]
Jul 14 19:36:52.701913 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:36:52.701873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:37:00.583319 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.580561    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"]
Jul 14 19:37:00.585713 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.585689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.588577 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.588552    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"sub-test-model-755a0b-kserve-self-signed-certs\""
Jul 14 19:37:00.596627 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.596598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"]
Jul 14 19:37:00.615991 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.615962    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:37:00.729745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.729699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tmp-dir\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.729929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.729825    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-home\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.729929 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.729846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-dshm\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.730032 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.729925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tls-certs\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.730032 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.729983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-model-cache\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.730032 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.730017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8rh\" (UniqueName: \"kubernetes.io/projected/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-kube-api-access-sc8rh\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831139 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-model-cache\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831139 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8rh\" (UniqueName: \"kubernetes.io/projected/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-kube-api-access-sc8rh\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tmp-dir\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-home\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-dshm\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tls-certs\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831597 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831577    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-model-cache\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831646 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tmp-dir\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.831683 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.831641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-home\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.833730 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.833671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-dshm\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.833941 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.833924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tls-certs\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.842721 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.842696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8rh\" (UniqueName: \"kubernetes.io/projected/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-kube-api-access-sc8rh\") pod \"sub-test-model-755a0b-kserve-5dbd9bd95-s9547\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") " pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:00.896756 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:00.896717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:01.039608 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:01.039575    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"]
Jul 14 19:37:01.042994 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:37:01.042966    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a1fda67_e0c5_4cff_a0d5_a3239443e8bd.slice/crio-495e4e2366786141c4f053c102a6c1b1965b56a12cf046b58419586971de1729 WatchSource:0}: Error finding container 495e4e2366786141c4f053c102a6c1b1965b56a12cf046b58419586971de1729: Status 404 returned error can't find the container with id 495e4e2366786141c4f053c102a6c1b1965b56a12cf046b58419586971de1729
Jul 14 19:37:01.639928 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:01.639889    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547" event={"ID":"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd","Type":"ContainerStarted","Data":"bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f"}
Jul 14 19:37:01.640321 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:01.639934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547" event={"ID":"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd","Type":"ContainerStarted","Data":"495e4e2366786141c4f053c102a6c1b1965b56a12cf046b58419586971de1729"}
Jul 14 19:37:01.661368 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:01.661305    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547" podStartSLOduration=1.661287696 podStartE2EDuration="1.661287696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:37:01.659147774 +0000 UTC m=+2565.531705994" watchObservedRunningTime="2026-07-14 19:37:01.661287696 +0000 UTC m=+2565.533845980"
Jul 14 19:37:06.852310 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:06.852260    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn" podUID="355312b8-77b6-4626-82ee-6880185141b1" containerName="main" containerID="cri-o://119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6" gracePeriod=45
Jul 14 19:37:07.109687 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.109615    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:37:07.186359 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186328    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355312b8-77b6-4626-82ee-6880185141b1-tls-certs\") pod \"355312b8-77b6-4626-82ee-6880185141b1\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") "
Jul 14 19:37:07.186543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-home\") pod \"355312b8-77b6-4626-82ee-6880185141b1\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") "
Jul 14 19:37:07.186543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186395    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-tmp-dir\") pod \"355312b8-77b6-4626-82ee-6880185141b1\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") "
Jul 14 19:37:07.186543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186438    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x22tc\" (UniqueName: \"kubernetes.io/projected/355312b8-77b6-4626-82ee-6880185141b1-kube-api-access-x22tc\") pod \"355312b8-77b6-4626-82ee-6880185141b1\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") "
Jul 14 19:37:07.186543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-model-cache\") pod \"355312b8-77b6-4626-82ee-6880185141b1\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") "
Jul 14 19:37:07.186543 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-dshm\") pod \"355312b8-77b6-4626-82ee-6880185141b1\" (UID: \"355312b8-77b6-4626-82ee-6880185141b1\") "
Jul 14 19:37:07.186809 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186655    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-home" (OuterVolumeSpecName: "home") pod "355312b8-77b6-4626-82ee-6880185141b1" (UID: "355312b8-77b6-4626-82ee-6880185141b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.186809 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186697    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "355312b8-77b6-4626-82ee-6880185141b1" (UID: "355312b8-77b6-4626-82ee-6880185141b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.186809 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186787    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "355312b8-77b6-4626-82ee-6880185141b1" (UID: "355312b8-77b6-4626-82ee-6880185141b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.186963 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186852    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.186963 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.186871    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.188863 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.188823    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-dshm" (OuterVolumeSpecName: "dshm") pod "355312b8-77b6-4626-82ee-6880185141b1" (UID: "355312b8-77b6-4626-82ee-6880185141b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.188863 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.188848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355312b8-77b6-4626-82ee-6880185141b1-kube-api-access-x22tc" (OuterVolumeSpecName: "kube-api-access-x22tc") pod "355312b8-77b6-4626-82ee-6880185141b1" (UID: "355312b8-77b6-4626-82ee-6880185141b1"). InnerVolumeSpecName "kube-api-access-x22tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:37:07.189010 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.188883    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355312b8-77b6-4626-82ee-6880185141b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "355312b8-77b6-4626-82ee-6880185141b1" (UID: "355312b8-77b6-4626-82ee-6880185141b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:37:07.288021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.287988    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22tc\" (UniqueName: \"kubernetes.io/projected/355312b8-77b6-4626-82ee-6880185141b1-kube-api-access-x22tc\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.288021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.288019    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.288021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.288029    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/355312b8-77b6-4626-82ee-6880185141b1-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.288250 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.288038    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/355312b8-77b6-4626-82ee-6880185141b1-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.672300 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.672263    2562 generic.go:356] "Generic (PLEG): container finished" podID="355312b8-77b6-4626-82ee-6880185141b1" containerID="119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6" exitCode=0
Jul 14 19:37:07.672491 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.672315    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"
Jul 14 19:37:07.672491 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.672354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn" event={"ID":"355312b8-77b6-4626-82ee-6880185141b1","Type":"ContainerDied","Data":"119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6"}
Jul 14 19:37:07.672491 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.672413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn" event={"ID":"355312b8-77b6-4626-82ee-6880185141b1","Type":"ContainerDied","Data":"c59f1171afdd5fb9d179c13ca2126396c892e20dcaaec2b877ed896a92cf9087"}
Jul 14 19:37:07.672491 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.672429    2562 scope.go:117] "RemoveContainer" containerID="119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6"
Jul 14 19:37:07.682228 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.682207    2562 scope.go:117] "RemoveContainer" containerID="119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6"
Jul 14 19:37:07.682513 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:37:07.682492    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6\": container with ID starting with 119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6 not found: ID does not exist" containerID="119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6"
Jul 14 19:37:07.682577 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.682522    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6"} err="failed to get container status \"119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6\": rpc error: code = NotFound desc = could not find container \"119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6\": container with ID starting with 119ad1935f1aa8afbcacbed3aa9e1e625d9d976c494afb904b88f3ce32902fd6 not found: ID does not exist"
Jul 14 19:37:07.697329 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.697302    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"]
Jul 14 19:37:07.700488 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.700467    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/auth-test-model-755a0b-kserve-6775688c95-8l2sn"]
Jul 14 19:37:07.709192 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.709159    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg" podUID="66c1bd37-b765-4a1a-8182-98f3774e51fb" containerName="main" containerID="cri-o://d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6" gracePeriod=45
Jul 14 19:37:07.964480 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.964455    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:37:07.994487 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-home\") pod \"66c1bd37-b765-4a1a-8182-98f3774e51fb\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") "
Jul 14 19:37:07.994487 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-tmp-dir\") pod \"66c1bd37-b765-4a1a-8182-98f3774e51fb\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") "
Jul 14 19:37:07.994708 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994571    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdkqn\" (UniqueName: \"kubernetes.io/projected/66c1bd37-b765-4a1a-8182-98f3774e51fb-kube-api-access-gdkqn\") pod \"66c1bd37-b765-4a1a-8182-98f3774e51fb\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") "
Jul 14 19:37:07.994708 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1bd37-b765-4a1a-8182-98f3774e51fb-tls-certs\") pod \"66c1bd37-b765-4a1a-8182-98f3774e51fb\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") "
Jul 14 19:37:07.994708 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-dshm\") pod \"66c1bd37-b765-4a1a-8182-98f3774e51fb\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") "
Jul 14 19:37:07.994858 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994729    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-model-cache\") pod \"66c1bd37-b765-4a1a-8182-98f3774e51fb\" (UID: \"66c1bd37-b765-4a1a-8182-98f3774e51fb\") "
Jul 14 19:37:07.994858 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994839    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66c1bd37-b765-4a1a-8182-98f3774e51fb" (UID: "66c1bd37-b765-4a1a-8182-98f3774e51fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.995044 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.994988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "66c1bd37-b765-4a1a-8182-98f3774e51fb" (UID: "66c1bd37-b765-4a1a-8182-98f3774e51fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.995185 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.995076    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-home" (OuterVolumeSpecName: "home") pod "66c1bd37-b765-4a1a-8182-98f3774e51fb" (UID: "66c1bd37-b765-4a1a-8182-98f3774e51fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.995185 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.995126    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.995185 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.995141    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:07.996933 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.996908    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c1bd37-b765-4a1a-8182-98f3774e51fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66c1bd37-b765-4a1a-8182-98f3774e51fb" (UID: "66c1bd37-b765-4a1a-8182-98f3774e51fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:37:07.997037 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.996943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-dshm" (OuterVolumeSpecName: "dshm") pod "66c1bd37-b765-4a1a-8182-98f3774e51fb" (UID: "66c1bd37-b765-4a1a-8182-98f3774e51fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:37:07.997203 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:07.997178    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c1bd37-b765-4a1a-8182-98f3774e51fb-kube-api-access-gdkqn" (OuterVolumeSpecName: "kube-api-access-gdkqn") pod "66c1bd37-b765-4a1a-8182-98f3774e51fb" (UID: "66c1bd37-b765-4a1a-8182-98f3774e51fb"). InnerVolumeSpecName "kube-api-access-gdkqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:37:08.096107 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.096064    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:08.096107 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.096099    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66c1bd37-b765-4a1a-8182-98f3774e51fb-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:08.096107 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.096108    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdkqn\" (UniqueName: \"kubernetes.io/projected/66c1bd37-b765-4a1a-8182-98f3774e51fb-kube-api-access-gdkqn\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:08.096107 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.096118    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66c1bd37-b765-4a1a-8182-98f3774e51fb-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:37:08.620795 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.620760    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="355312b8-77b6-4626-82ee-6880185141b1" path="/var/lib/kubelet/pods/355312b8-77b6-4626-82ee-6880185141b1/volumes"
Jul 14 19:37:08.678567 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.678529    2562 generic.go:356] "Generic (PLEG): container finished" podID="66c1bd37-b765-4a1a-8182-98f3774e51fb" containerID="d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6" exitCode=0
Jul 14 19:37:08.678742 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.678578    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"
Jul 14 19:37:08.678742 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.678605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg" event={"ID":"66c1bd37-b765-4a1a-8182-98f3774e51fb","Type":"ContainerDied","Data":"d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6"}
Jul 14 19:37:08.678742 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.678643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg" event={"ID":"66c1bd37-b765-4a1a-8182-98f3774e51fb","Type":"ContainerDied","Data":"7cc7de06b499591f6f2af09c79f886229fd50c3c42536e98a72e80ef9af65028"}
Jul 14 19:37:08.678742 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.678662    2562 scope.go:117] "RemoveContainer" containerID="d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6"
Jul 14 19:37:08.687415 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.687385    2562 scope.go:117] "RemoveContainer" containerID="d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6"
Jul 14 19:37:08.687729 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:37:08.687709    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6\": container with ID starting with d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6 not found: ID does not exist" containerID="d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6"
Jul 14 19:37:08.687797 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.687736    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6"} err="failed to get container status \"d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6\": rpc error: code = NotFound desc = could not find container \"d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6\": container with ID starting with d7551966235dff52e83f5c737eae14b022ec6a731da08c3d2e9dcad69dca46e6 not found: ID does not exist"
Jul 14 19:37:08.701582 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.701544    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"]
Jul 14 19:37:08.706163 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:08.706127    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/auth-test-model-7c2fbe-kserve-5f588598d7-wvhdg"]
Jul 14 19:37:10.621478 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:10.621444    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c1bd37-b765-4a1a-8182-98f3774e51fb" path="/var/lib/kubelet/pods/66c1bd37-b765-4a1a-8182-98f3774e51fb/volumes"
Jul 14 19:37:10.897459 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:10.897343    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:10.897459 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:10.897392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:10.910010 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:10.909983    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:11.707347 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:11.707316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:37:12.049289 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:12.049256    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:37:12.246974 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:12.246941    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:37:29.290129 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290096    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"]
Jul 14 19:37:29.290626 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290482    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c1bd37-b765-4a1a-8182-98f3774e51fb" containerName="main"
Jul 14 19:37:29.290626 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290495    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c1bd37-b765-4a1a-8182-98f3774e51fb" containerName="main"
Jul 14 19:37:29.290626 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290511    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="355312b8-77b6-4626-82ee-6880185141b1" containerName="main"
Jul 14 19:37:29.290626 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290516    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="355312b8-77b6-4626-82ee-6880185141b1" containerName="main"
Jul 14 19:37:29.290626 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290579    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c1bd37-b765-4a1a-8182-98f3774e51fb" containerName="main"
Jul 14 19:37:29.290626 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.290588    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="355312b8-77b6-4626-82ee-6880185141b1" containerName="main"
Jul 14 19:37:29.293786 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.293770    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.297871 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.297847    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"kube-root-ca.crt\""
Jul 14 19:37:29.299056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.299037    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"default-dockercfg-xphld\""
Jul 14 19:37:29.299056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.299044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"sub-test-model-7c2fbe-kserve-self-signed-certs\""
Jul 14 19:37:29.299239 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.299074    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"openshift-service-ca.crt\""
Jul 14 19:37:29.314741 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.314707    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"]
Jul 14 19:37:29.332241 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.332199    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:37:29.390946 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.390903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-model-cache\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.390946 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.390949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-home\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.391170 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.391052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8r8v\" (UniqueName: \"kubernetes.io/projected/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-kube-api-access-l8r8v\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.391170 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.391084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tls-certs\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.391170 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.391150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-dshm\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.391170 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.391169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tmp-dir\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.491808 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.491762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8r8v\" (UniqueName: \"kubernetes.io/projected/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-kube-api-access-l8r8v\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.491808 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.491809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tls-certs\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492069 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.491860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-dshm\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492069 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.491883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tmp-dir\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492069 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.491936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-model-cache\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492069 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.491974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-home\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492438 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.492380    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-model-cache\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492438 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.492394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tmp-dir\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.492559 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.492459    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-home\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.494362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.494336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-dshm\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.494615 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.494600    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tls-certs\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.503345 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.503325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8r8v\" (UniqueName: \"kubernetes.io/projected/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-kube-api-access-l8r8v\") pod \"sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") " pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.604988 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.604873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:29.749780 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.749752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"]
Jul 14 19:37:29.750517 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:37:29.750491    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef82872_a2c6_4e0e_8eb0_a0a2d462ac53.slice/crio-004195a49ca2fbd8106ac0422419df6dcc0542525eaee204e4f6f55ef94a798f WatchSource:0}: Error finding container 004195a49ca2fbd8106ac0422419df6dcc0542525eaee204e4f6f55ef94a798f: Status 404 returned error can't find the container with id 004195a49ca2fbd8106ac0422419df6dcc0542525eaee204e4f6f55ef94a798f
Jul 14 19:37:29.794097 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:29.794056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km" event={"ID":"def82872-a2c6-4e0e-8eb0-a0a2d462ac53","Type":"ContainerStarted","Data":"004195a49ca2fbd8106ac0422419df6dcc0542525eaee204e4f6f55ef94a798f"}
Jul 14 19:37:30.801050 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:30.801005    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km" event={"ID":"def82872-a2c6-4e0e-8eb0-a0a2d462ac53","Type":"ContainerStarted","Data":"b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808"}
Jul 14 19:37:30.822769 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:30.822714    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km" podStartSLOduration=1.822698902 podStartE2EDuration="1.822698902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:37:30.820997608 +0000 UTC m=+2594.693555830" watchObservedRunningTime="2026-07-14 19:37:30.822698902 +0000 UTC m=+2594.695257152"
Jul 14 19:37:39.606031 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:39.605936    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:39.606031 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:39.606011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:39.618564 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:39.618533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:37:39.859581 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:37:39.859501    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:38:35.439310 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:35.439272    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"]
Jul 14 19:38:36.454213 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:36.454177    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"]
Jul 14 19:38:37.690536 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:37.690498    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:38:39.950279 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:39.950233    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:38:42.808486 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:42.808420    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:38:50.459503 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.459458    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547" podUID="6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" containerName="main" containerID="cri-o://bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f" gracePeriod=45
Jul 14 19:38:50.711032 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.710964    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:38:50.754250 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754218    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-dshm\") pod \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") "
Jul 14 19:38:50.754250 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754253    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-model-cache\") pod \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") "
Jul 14 19:38:50.754550 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc8rh\" (UniqueName: \"kubernetes.io/projected/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-kube-api-access-sc8rh\") pod \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") "
Jul 14 19:38:50.754550 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-home\") pod \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") "
Jul 14 19:38:50.754550 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tls-certs\") pod \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") "
Jul 14 19:38:50.754550 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754452    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tmp-dir\") pod \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\" (UID: \"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd\") "
Jul 14 19:38:50.754756 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754569    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" (UID: "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:50.754811 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754757    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" (UID: "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:50.754865 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754814    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:50.754910 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.754862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-home" (OuterVolumeSpecName: "home") pod "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" (UID: "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:50.756502 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.756478    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-dshm" (OuterVolumeSpecName: "dshm") pod "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" (UID: "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:50.756675 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.756651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" (UID: "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:38:50.756951 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.756935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-kube-api-access-sc8rh" (OuterVolumeSpecName: "kube-api-access-sc8rh") pod "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" (UID: "6a1fda67-e0c5-4cff-a0d5-a3239443e8bd"). InnerVolumeSpecName "kube-api-access-sc8rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:38:50.855309 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.855268    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:50.855309 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.855303    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8rh\" (UniqueName: \"kubernetes.io/projected/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-kube-api-access-sc8rh\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:50.855309 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.855320    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:50.855592 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.855329    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:50.855592 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:50.855337    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:51.221155 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.221117    2562 generic.go:356] "Generic (PLEG): container finished" podID="6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" containerID="bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f" exitCode=0
Jul 14 19:38:51.221358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.221190    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"
Jul 14 19:38:51.221358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.221206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547" event={"ID":"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd","Type":"ContainerDied","Data":"bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f"}
Jul 14 19:38:51.221358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.221253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547" event={"ID":"6a1fda67-e0c5-4cff-a0d5-a3239443e8bd","Type":"ContainerDied","Data":"495e4e2366786141c4f053c102a6c1b1965b56a12cf046b58419586971de1729"}
Jul 14 19:38:51.221358 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.221271    2562 scope.go:117] "RemoveContainer" containerID="bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f"
Jul 14 19:38:51.231021 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.231003    2562 scope.go:117] "RemoveContainer" containerID="bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f"
Jul 14 19:38:51.231272 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:38:51.231255    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f\": container with ID starting with bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f not found: ID does not exist" containerID="bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f"
Jul 14 19:38:51.231318 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.231280    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f"} err="failed to get container status \"bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f\": rpc error: code = NotFound desc = could not find container \"bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f\": container with ID starting with bc69dee6f73e1048749ba416098e2770d8a8cd16645c7922f9af8afb11e91b2f not found: ID does not exist"
Jul 14 19:38:51.243631 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.243605    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"]
Jul 14 19:38:51.248653 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.248628    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/sub-test-model-755a0b-kserve-5dbd9bd95-s9547"]
Jul 14 19:38:51.474529 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.474392    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km" podUID="def82872-a2c6-4e0e-8eb0-a0a2d462ac53" containerName="main" containerID="cri-o://b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808" gracePeriod=45
Jul 14 19:38:51.732885 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.732817    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:38:51.762639 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-home\") pod \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") "
Jul 14 19:38:51.762639 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-dshm\") pod \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") "
Jul 14 19:38:51.762945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-model-cache\") pod \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") "
Jul 14 19:38:51.762945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762710    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8r8v\" (UniqueName: \"kubernetes.io/projected/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-kube-api-access-l8r8v\") pod \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") "
Jul 14 19:38:51.762945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762728    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tls-certs\") pod \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") "
Jul 14 19:38:51.762945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762765    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tmp-dir\") pod \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\" (UID: \"def82872-a2c6-4e0e-8eb0-a0a2d462ac53\") "
Jul 14 19:38:51.762945 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.762929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-home" (OuterVolumeSpecName: "home") pod "def82872-a2c6-4e0e-8eb0-a0a2d462ac53" (UID: "def82872-a2c6-4e0e-8eb0-a0a2d462ac53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:51.763207 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.763083    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:51.763260 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.763190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-model-cache" (OuterVolumeSpecName: "model-cache") pod "def82872-a2c6-4e0e-8eb0-a0a2d462ac53" (UID: "def82872-a2c6-4e0e-8eb0-a0a2d462ac53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:51.763260 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.763209    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "def82872-a2c6-4e0e-8eb0-a0a2d462ac53" (UID: "def82872-a2c6-4e0e-8eb0-a0a2d462ac53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:51.765628 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.765598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-dshm" (OuterVolumeSpecName: "dshm") pod "def82872-a2c6-4e0e-8eb0-a0a2d462ac53" (UID: "def82872-a2c6-4e0e-8eb0-a0a2d462ac53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:38:51.765628 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.765616    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "def82872-a2c6-4e0e-8eb0-a0a2d462ac53" (UID: "def82872-a2c6-4e0e-8eb0-a0a2d462ac53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:38:51.765804 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.765700    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-kube-api-access-l8r8v" (OuterVolumeSpecName: "kube-api-access-l8r8v") pod "def82872-a2c6-4e0e-8eb0-a0a2d462ac53" (UID: "def82872-a2c6-4e0e-8eb0-a0a2d462ac53"). InnerVolumeSpecName "kube-api-access-l8r8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:38:51.864150 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.864104    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:51.864150 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.864140    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:51.864150 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.864151    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8r8v\" (UniqueName: \"kubernetes.io/projected/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-kube-api-access-l8r8v\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:51.864150 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.864160    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:51.864461 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:51.864190    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def82872-a2c6-4e0e-8eb0-a0a2d462ac53-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:38:52.228417 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.228362    2562 generic.go:356] "Generic (PLEG): container finished" podID="def82872-a2c6-4e0e-8eb0-a0a2d462ac53" containerID="b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808" exitCode=0
Jul 14 19:38:52.228593 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.228432    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"
Jul 14 19:38:52.228593 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.228444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km" event={"ID":"def82872-a2c6-4e0e-8eb0-a0a2d462ac53","Type":"ContainerDied","Data":"b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808"}
Jul 14 19:38:52.228593 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.228491    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km" event={"ID":"def82872-a2c6-4e0e-8eb0-a0a2d462ac53","Type":"ContainerDied","Data":"004195a49ca2fbd8106ac0422419df6dcc0542525eaee204e4f6f55ef94a798f"}
Jul 14 19:38:52.228593 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.228512    2562 scope.go:117] "RemoveContainer" containerID="b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808"
Jul 14 19:38:52.239947 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.239928    2562 scope.go:117] "RemoveContainer" containerID="b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808"
Jul 14 19:38:52.240250 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:38:52.240231    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808\": container with ID starting with b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808 not found: ID does not exist" containerID="b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808"
Jul 14 19:38:52.240300 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.240258    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808"} err="failed to get container status \"b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808\": rpc error: code = NotFound desc = could not find container \"b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808\": container with ID starting with b6994f51e2b286e8dde867ca81dc05a10a14d47f318a3af7c6ec0bd2900ab808 not found: ID does not exist"
Jul 14 19:38:52.253242 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.253211    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"]
Jul 14 19:38:52.258105 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.258079    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/sub-test-model-7c2fbe-kserve-6f88d79db9-dc2km"]
Jul 14 19:38:52.622612 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.622535    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" path="/var/lib/kubelet/pods/6a1fda67-e0c5-4cff-a0d5-a3239443e8bd/volumes"
Jul 14 19:38:52.622965 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.622945    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def82872-a2c6-4e0e-8eb0-a0a2d462ac53" path="/var/lib/kubelet/pods/def82872-a2c6-4e0e-8eb0-a0a2d462ac53/volumes"
Jul 14 19:38:52.899826 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.899748    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"]
Jul 14 19:38:52.900195 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.900179    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def82872-a2c6-4e0e-8eb0-a0a2d462ac53" containerName="main"
Jul 14 19:38:52.900244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.900199    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="def82872-a2c6-4e0e-8eb0-a0a2d462ac53" containerName="main"
Jul 14 19:38:52.900244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.900212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" containerName="main"
Jul 14 19:38:52.900244 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.900217    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" containerName="main"
Jul 14 19:38:52.900340 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.900279    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a1fda67-e0c5-4cff-a0d5-a3239443e8bd" containerName="main"
Jul 14 19:38:52.900340 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.900286    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="def82872-a2c6-4e0e-8eb0-a0a2d462ac53" containerName="main"
Jul 14 19:38:52.904513 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.904497    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:52.909991 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.909967    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"kube-root-ca.crt\""
Jul 14 19:38:52.911282 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.911154    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"openshift-service-ca.crt\""
Jul 14 19:38:52.911282 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.911154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"default-dockercfg-8j764\""
Jul 14 19:38:52.911518 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.911506    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-755a0b\"/\"rate-test-model-755a0b-kserve-self-signed-certs\""
Jul 14 19:38:52.917662 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.917636    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"]
Jul 14 19:38:52.949584 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.949558    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:38:52.973853 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.973825    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-dshm\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:52.974000 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.973860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tmp-dir\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:52.974000 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.973889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-model-cache\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:52.974000 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.973939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-home\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:52.974111 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.974003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qsp\" (UniqueName: \"kubernetes.io/projected/c185b1a9-d771-4c7e-84e7-724f41e5bff1-kube-api-access-q7qsp\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:52.974111 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:52.974059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tls-certs\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.075526 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.075474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tls-certs\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.075722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.075557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-dshm\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.075722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.075586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tmp-dir\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.075722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.075615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-model-cache\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.075722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.075647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-home\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.075722 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.075679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qsp\" (UniqueName: \"kubernetes.io/projected/c185b1a9-d771-4c7e-84e7-724f41e5bff1-kube-api-access-q7qsp\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.076088 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.076054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-model-cache\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.076196 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.076096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tmp-dir\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.076196 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.076148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-home\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.078099 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.078073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-dshm\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.078337 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.078318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tls-certs\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.086134 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.086110    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qsp\" (UniqueName: \"kubernetes.io/projected/c185b1a9-d771-4c7e-84e7-724f41e5bff1-kube-api-access-q7qsp\") pod \"rate-test-model-755a0b-kserve-6b68dc679-jlt5q\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") " pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.216556 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.216513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:38:53.357284 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.357258    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"]
Jul 14 19:38:53.358875 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:38:53.358847    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc185b1a9_d771_4c7e_84e7_724f41e5bff1.slice/crio-9de23d18f116c3af301e260bbab48a384b9b41949d3f4b7af183bc7eedc97226 WatchSource:0}: Error finding container 9de23d18f116c3af301e260bbab48a384b9b41949d3f4b7af183bc7eedc97226: Status 404 returned error can't find the container with id 9de23d18f116c3af301e260bbab48a384b9b41949d3f4b7af183bc7eedc97226
Jul 14 19:38:53.408505 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:53.408473    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:38:54.244952 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:54.244918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q" event={"ID":"c185b1a9-d771-4c7e-84e7-724f41e5bff1","Type":"ContainerStarted","Data":"a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080"}
Jul 14 19:38:54.244952 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:54.244954    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q" event={"ID":"c185b1a9-d771-4c7e-84e7-724f41e5bff1","Type":"ContainerStarted","Data":"9de23d18f116c3af301e260bbab48a384b9b41949d3f4b7af183bc7eedc97226"}
Jul 14 19:38:54.265487 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:38:54.265435    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q" podStartSLOduration=2.265418536 podStartE2EDuration="2.265418536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:38:54.264031066 +0000 UTC m=+2678.136589285" watchObservedRunningTime="2026-07-14 19:38:54.265418536 +0000 UTC m=+2678.137976747"
Jul 14 19:39:03.216975 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:03.216934    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:39:03.217387 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:03.217094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:39:03.229878 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:03.229852    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:39:03.301278 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:03.301243    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:39:14.757690 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:14.757602    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:39:16.866225 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:16.866193    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:39:16.964708 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:16.964670    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:39:26.106618 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:26.106583    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:39:26.207276 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:26.207245    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:39:39.393890 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.393852    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"]
Jul 14 19:39:39.397493 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.397474    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.402555 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.402521    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"kube-root-ca.crt\""
Jul 14 19:39:39.405788 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.405757    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"rate-test-model-7c2fbe-kserve-self-signed-certs\""
Jul 14 19:39:39.405901 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.405842    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"default-dockercfg-xphld\""
Jul 14 19:39:39.406200 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.406183    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7c2fbe\"/\"openshift-service-ca.crt\""
Jul 14 19:39:39.416806 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.416774    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"]
Jul 14 19:39:39.454197 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.454160    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:39:39.511479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.509651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-dshm\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.511479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.509719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tls-certs\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.511479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.509770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-model-cache\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.511479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.509801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28ld\" (UniqueName: \"kubernetes.io/projected/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-kube-api-access-d28ld\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.511479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.509839    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tmp-dir\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.511479 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.509914    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-home\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.610604 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.610563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-model-cache\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.610604 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.610604    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28ld\" (UniqueName: \"kubernetes.io/projected/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-kube-api-access-d28ld\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.610852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.610628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tmp-dir\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.610852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.610679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-home\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.610852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.610752    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-dshm\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.610852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.610776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tls-certs\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.611088 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.611061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-model-cache\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.611157 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.611130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tmp-dir\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.611157 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.611148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-home\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.613241 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.613212    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-dshm\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.613470 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.613451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tls-certs\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.630424 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.630374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28ld\" (UniqueName: \"kubernetes.io/projected/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-kube-api-access-d28ld\") pod \"rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") " pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.707575 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.707499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:39.860294 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:39.860269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"]
Jul 14 19:39:39.861603 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:39:39.861580    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f6806a7_3de6_49d4_9bdf_d45a686ce91e.slice/crio-c178ffc7834245dac7c0237c4d6f8928156178c43da977a8b01cfb30b534d447 WatchSource:0}: Error finding container c178ffc7834245dac7c0237c4d6f8928156178c43da977a8b01cfb30b534d447: Status 404 returned error can't find the container with id c178ffc7834245dac7c0237c4d6f8928156178c43da977a8b01cfb30b534d447
Jul 14 19:39:40.483246 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:40.483202    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr" event={"ID":"7f6806a7-3de6-49d4-9bdf-d45a686ce91e","Type":"ContainerStarted","Data":"e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba"}
Jul 14 19:39:40.483246 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:40.483253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr" event={"ID":"7f6806a7-3de6-49d4-9bdf-d45a686ce91e","Type":"ContainerStarted","Data":"c178ffc7834245dac7c0237c4d6f8928156178c43da977a8b01cfb30b534d447"}
Jul 14 19:39:40.506799 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:40.506744    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr" podStartSLOduration=1.50672612 podStartE2EDuration="1.50672612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:39:40.504644769 +0000 UTC m=+2724.377202999" watchObservedRunningTime="2026-07-14 19:39:40.50672612 +0000 UTC m=+2724.379284340"
Jul 14 19:39:49.708100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:49.708065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:49.708649 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:49.708133    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:49.721309 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:49.721283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:39:50.543236 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:39:50.543207    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:40:26.882624 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:26.882578    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"]
Jul 14 19:40:26.916570 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:26.916541    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:40:27.784579 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:27.784548    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"]
Jul 14 19:40:31.006304 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:31.006264    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:40:41.902868 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:41.902780    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q" podUID="c185b1a9-d771-4c7e-84e7-724f41e5bff1" containerName="main" containerID="cri-o://a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080" gracePeriod=45
Jul 14 19:40:42.149986 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.149962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:40:42.289855 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.289815    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tmp-dir\") pod \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") "
Jul 14 19:40:42.290071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.289878    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7qsp\" (UniqueName: \"kubernetes.io/projected/c185b1a9-d771-4c7e-84e7-724f41e5bff1-kube-api-access-q7qsp\") pod \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") "
Jul 14 19:40:42.290071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.289925    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-dshm\") pod \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") "
Jul 14 19:40:42.290071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.289940    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-home\") pod \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") "
Jul 14 19:40:42.290071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.289966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-model-cache\") pod \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") "
Jul 14 19:40:42.290071 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.290005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tls-certs\") pod \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\" (UID: \"c185b1a9-d771-4c7e-84e7-724f41e5bff1\") "
Jul 14 19:40:42.290392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.290160    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c185b1a9-d771-4c7e-84e7-724f41e5bff1" (UID: "c185b1a9-d771-4c7e-84e7-724f41e5bff1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:42.290392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.290234    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-home" (OuterVolumeSpecName: "home") pod "c185b1a9-d771-4c7e-84e7-724f41e5bff1" (UID: "c185b1a9-d771-4c7e-84e7-724f41e5bff1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:42.290392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.290251    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c185b1a9-d771-4c7e-84e7-724f41e5bff1" (UID: "c185b1a9-d771-4c7e-84e7-724f41e5bff1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:42.292289 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.292260    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c185b1a9-d771-4c7e-84e7-724f41e5bff1" (UID: "c185b1a9-d771-4c7e-84e7-724f41e5bff1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:40:42.292392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.292301    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c185b1a9-d771-4c7e-84e7-724f41e5bff1-kube-api-access-q7qsp" (OuterVolumeSpecName: "kube-api-access-q7qsp") pod "c185b1a9-d771-4c7e-84e7-724f41e5bff1" (UID: "c185b1a9-d771-4c7e-84e7-724f41e5bff1"). InnerVolumeSpecName "kube-api-access-q7qsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:40:42.292392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.292300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-dshm" (OuterVolumeSpecName: "dshm") pod "c185b1a9-d771-4c7e-84e7-724f41e5bff1" (UID: "c185b1a9-d771-4c7e-84e7-724f41e5bff1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:42.391586 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.391536    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7qsp\" (UniqueName: \"kubernetes.io/projected/c185b1a9-d771-4c7e-84e7-724f41e5bff1-kube-api-access-q7qsp\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:42.391586 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.391581    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:42.391586 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.391592    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:42.391586 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.391600    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:42.391858 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.391609    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:42.391858 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.391616    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c185b1a9-d771-4c7e-84e7-724f41e5bff1-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:42.796692 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.796650    2562 generic.go:356] "Generic (PLEG): container finished" podID="c185b1a9-d771-4c7e-84e7-724f41e5bff1" containerID="a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080" exitCode=0
Jul 14 19:40:42.796887 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.796788    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q" event={"ID":"c185b1a9-d771-4c7e-84e7-724f41e5bff1","Type":"ContainerDied","Data":"a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080"}
Jul 14 19:40:42.796887 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.796826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q" event={"ID":"c185b1a9-d771-4c7e-84e7-724f41e5bff1","Type":"ContainerDied","Data":"9de23d18f116c3af301e260bbab48a384b9b41949d3f4b7af183bc7eedc97226"}
Jul 14 19:40:42.796887 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.796858    2562 scope.go:117] "RemoveContainer" containerID="a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080"
Jul 14 19:40:42.797124 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.797061    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"
Jul 14 19:40:42.806165 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.806112    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr" podUID="7f6806a7-3de6-49d4-9bdf-d45a686ce91e" containerName="main" containerID="cri-o://e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba" gracePeriod=45
Jul 14 19:40:42.813338 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.813317    2562 scope.go:117] "RemoveContainer" containerID="a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080"
Jul 14 19:40:42.813717 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:40:42.813689    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080\": container with ID starting with a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080 not found: ID does not exist" containerID="a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080"
Jul 14 19:40:42.813795 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.813727    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080"} err="failed to get container status \"a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080\": rpc error: code = NotFound desc = could not find container \"a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080\": container with ID starting with a6823b40131588e00f7cbf448fcf270771892c9032475c6509a87269dfb14080 not found: ID does not exist"
Jul 14 19:40:42.822314 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.822291    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"]
Jul 14 19:40:42.825392 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:42.825371    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-755a0b/rate-test-model-755a0b-kserve-6b68dc679-jlt5q"]
Jul 14 19:40:43.065100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.065071    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:40:43.199039 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.198937    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d28ld\" (UniqueName: \"kubernetes.io/projected/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-kube-api-access-d28ld\") pod \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") "
Jul 14 19:40:43.199039 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199036    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-home\") pod \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") "
Jul 14 19:40:43.199241 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199147    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-model-cache\") pod \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") "
Jul 14 19:40:43.199241 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199188    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tmp-dir\") pod \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") "
Jul 14 19:40:43.199356 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199255    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tls-certs\") pod \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") "
Jul 14 19:40:43.199356 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199289    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-home" (OuterVolumeSpecName: "home") pod "7f6806a7-3de6-49d4-9bdf-d45a686ce91e" (UID: "7f6806a7-3de6-49d4-9bdf-d45a686ce91e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:43.199356 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199332    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-dshm\") pod \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\" (UID: \"7f6806a7-3de6-49d4-9bdf-d45a686ce91e\") "
Jul 14 19:40:43.199559 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199458    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f6806a7-3de6-49d4-9bdf-d45a686ce91e" (UID: "7f6806a7-3de6-49d4-9bdf-d45a686ce91e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:43.199559 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f6806a7-3de6-49d4-9bdf-d45a686ce91e" (UID: "7f6806a7-3de6-49d4-9bdf-d45a686ce91e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:43.199702 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199681    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-home\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:43.199760 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199708    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-model-cache\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:43.199760 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.199748    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tmp-dir\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:43.201270 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.201248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-kube-api-access-d28ld" (OuterVolumeSpecName: "kube-api-access-d28ld") pod "7f6806a7-3de6-49d4-9bdf-d45a686ce91e" (UID: "7f6806a7-3de6-49d4-9bdf-d45a686ce91e"). InnerVolumeSpecName "kube-api-access-d28ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:40:43.201510 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.201495    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f6806a7-3de6-49d4-9bdf-d45a686ce91e" (UID: "7f6806a7-3de6-49d4-9bdf-d45a686ce91e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:40:43.201618 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.201606    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-dshm" (OuterVolumeSpecName: "dshm") pod "7f6806a7-3de6-49d4-9bdf-d45a686ce91e" (UID: "7f6806a7-3de6-49d4-9bdf-d45a686ce91e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 19:40:43.300228 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.300192    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d28ld\" (UniqueName: \"kubernetes.io/projected/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-kube-api-access-d28ld\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:43.300228 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.300224    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-tls-certs\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:43.300228 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.300235    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6806a7-3de6-49d4-9bdf-d45a686ce91e-dshm\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:43.804130 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.804097    2562 generic.go:356] "Generic (PLEG): container finished" podID="7f6806a7-3de6-49d4-9bdf-d45a686ce91e" containerID="e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba" exitCode=0
Jul 14 19:40:43.804316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.804137    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"
Jul 14 19:40:43.804316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.804155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr" event={"ID":"7f6806a7-3de6-49d4-9bdf-d45a686ce91e","Type":"ContainerDied","Data":"e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba"}
Jul 14 19:40:43.804316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.804184    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr" event={"ID":"7f6806a7-3de6-49d4-9bdf-d45a686ce91e","Type":"ContainerDied","Data":"c178ffc7834245dac7c0237c4d6f8928156178c43da977a8b01cfb30b534d447"}
Jul 14 19:40:43.804316 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.804201    2562 scope.go:117] "RemoveContainer" containerID="e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba"
Jul 14 19:40:43.813992 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.813971    2562 scope.go:117] "RemoveContainer" containerID="e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba"
Jul 14 19:40:43.814251 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:40:43.814235    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba\": container with ID starting with e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba not found: ID does not exist" containerID="e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba"
Jul 14 19:40:43.814299 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.814260    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba"} err="failed to get container status \"e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba\": rpc error: code = NotFound desc = could not find container \"e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba\": container with ID starting with e6ccbdf5a6fb7670f4ff93ba1e9ecfa397631deb252834d87f03f82147ad8eba not found: ID does not exist"
Jul 14 19:40:43.829214 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.829189    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"]
Jul 14 19:40:43.833911 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:43.833887    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7c2fbe/rate-test-model-7c2fbe-kserve-7dcfc684b7-xcqtr"]
Jul 14 19:40:44.621937 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:44.621905    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f6806a7-3de6-49d4-9bdf-d45a686ce91e" path="/var/lib/kubelet/pods/7f6806a7-3de6-49d4-9bdf-d45a686ce91e/volumes"
Jul 14 19:40:44.622329 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:44.622286    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c185b1a9-d771-4c7e-84e7-724f41e5bff1" path="/var/lib/kubelet/pods/c185b1a9-d771-4c7e-84e7-724f41e5bff1/volumes"
Jul 14 19:40:47.147344 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147310    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"]
Jul 14 19:40:47.147739 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147703    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c185b1a9-d771-4c7e-84e7-724f41e5bff1" containerName="main"
Jul 14 19:40:47.147739 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147714    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c185b1a9-d771-4c7e-84e7-724f41e5bff1" containerName="main"
Jul 14 19:40:47.147739 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147733    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f6806a7-3de6-49d4-9bdf-d45a686ce91e" containerName="main"
Jul 14 19:40:47.147739 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147738    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f6806a7-3de6-49d4-9bdf-d45a686ce91e" containerName="main"
Jul 14 19:40:47.148100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147795    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f6806a7-3de6-49d4-9bdf-d45a686ce91e" containerName="main"
Jul 14 19:40:47.148100 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.147803    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c185b1a9-d771-4c7e-84e7-724f41e5bff1" containerName="main"
Jul 14 19:40:47.151880 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.151862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:47.160000 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.159982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:40:47.168981 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.168956    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"]
Jul 14 19:40:47.338196 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.338165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ffe92b-96ad-4715-bc4d-e4058d33a69d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m\" (UID: \"37ffe92b-96ad-4715-bc4d-e4058d33a69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:47.439301 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.439210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ffe92b-96ad-4715-bc4d-e4058d33a69d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m\" (UID: \"37ffe92b-96ad-4715-bc4d-e4058d33a69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:47.439852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.439834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ffe92b-96ad-4715-bc4d-e4058d33a69d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m\" (UID: \"37ffe92b-96ad-4715-bc4d-e4058d33a69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:47.461260 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.461231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:47.607794 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.607766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"]
Jul 14 19:40:47.609814 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:40:47.609787    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ffe92b_96ad_4715_bc4d_e4058d33a69d.slice/crio-c59485da7a56553587ec21a9befdb57d3dcbfee23b12a743dd9512932e099bfe WatchSource:0}: Error finding container c59485da7a56553587ec21a9befdb57d3dcbfee23b12a743dd9512932e099bfe: Status 404 returned error can't find the container with id c59485da7a56553587ec21a9befdb57d3dcbfee23b12a743dd9512932e099bfe
Jul 14 19:40:47.826089 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.826057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m" event={"ID":"37ffe92b-96ad-4715-bc4d-e4058d33a69d","Type":"ContainerStarted","Data":"c59485da7a56553587ec21a9befdb57d3dcbfee23b12a743dd9512932e099bfe"}
Jul 14 19:40:47.943178 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:47.943137    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:40:48.043727 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:48.043678    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:40:48.832894 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:48.832855    2562 generic.go:356] "Generic (PLEG): container finished" podID="37ffe92b-96ad-4715-bc4d-e4058d33a69d" containerID="00fe78704f31ad3f8e09bbe7d7905a585a22cb1be83da7edbdc25fee5a8aacf7" exitCode=0
Jul 14 19:40:48.833273 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:48.832908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m" event={"ID":"37ffe92b-96ad-4715-bc4d-e4058d33a69d","Type":"ContainerDied","Data":"00fe78704f31ad3f8e09bbe7d7905a585a22cb1be83da7edbdc25fee5a8aacf7"}
Jul 14 19:40:49.970224 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:49.970194    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:50.164252 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:50.164140    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ffe92b-96ad-4715-bc4d-e4058d33a69d-maas-api-service-ca\") pod \"37ffe92b-96ad-4715-bc4d-e4058d33a69d\" (UID: \"37ffe92b-96ad-4715-bc4d-e4058d33a69d\") "
Jul 14 19:40:50.164567 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:50.164545    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ffe92b-96ad-4715-bc4d-e4058d33a69d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37ffe92b-96ad-4715-bc4d-e4058d33a69d" (UID: "37ffe92b-96ad-4715-bc4d-e4058d33a69d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:40:50.264910 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:50.264879    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37ffe92b-96ad-4715-bc4d-e4058d33a69d-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:50.844709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:50.844676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m" event={"ID":"37ffe92b-96ad-4715-bc4d-e4058d33a69d","Type":"ContainerDied","Data":"c59485da7a56553587ec21a9befdb57d3dcbfee23b12a743dd9512932e099bfe"}
Jul 14 19:40:50.844709 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:50.844701    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m"
Jul 14 19:40:50.844993 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:50.844727    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59485da7a56553587ec21a9befdb57d3dcbfee23b12a743dd9512932e099bfe"
Jul 14 19:40:57.873569 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:57.873538    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"]
Jul 14 19:40:57.874070 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:57.873787    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" podUID="c402efd7-2d02-45e4-a94c-e65e09da4c39" containerName="maas-api" containerID="cri-o://c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c" gracePeriod=30
Jul 14 19:40:57.905920 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:57.905887    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:40:58.123505 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.123476    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:40:58.224514 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.224482    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c402efd7-2d02-45e4-a94c-e65e09da4c39-openshift-service-ca\") pod \"c402efd7-2d02-45e4-a94c-e65e09da4c39\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") "
Jul 14 19:40:58.224710 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.224530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rpcn\" (UniqueName: \"kubernetes.io/projected/c402efd7-2d02-45e4-a94c-e65e09da4c39-kube-api-access-4rpcn\") pod \"c402efd7-2d02-45e4-a94c-e65e09da4c39\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") "
Jul 14 19:40:58.224710 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.224584    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls\") pod \"c402efd7-2d02-45e4-a94c-e65e09da4c39\" (UID: \"c402efd7-2d02-45e4-a94c-e65e09da4c39\") "
Jul 14 19:40:58.224872 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.224848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c402efd7-2d02-45e4-a94c-e65e09da4c39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c402efd7-2d02-45e4-a94c-e65e09da4c39" (UID: "c402efd7-2d02-45e4-a94c-e65e09da4c39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:40:58.226755 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.226725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c402efd7-2d02-45e4-a94c-e65e09da4c39" (UID: "c402efd7-2d02-45e4-a94c-e65e09da4c39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:40:58.226907 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.226889    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c402efd7-2d02-45e4-a94c-e65e09da4c39-kube-api-access-4rpcn" (OuterVolumeSpecName: "kube-api-access-4rpcn") pod "c402efd7-2d02-45e4-a94c-e65e09da4c39" (UID: "c402efd7-2d02-45e4-a94c-e65e09da4c39"). InnerVolumeSpecName "kube-api-access-4rpcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:40:58.326061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.326024    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c402efd7-2d02-45e4-a94c-e65e09da4c39-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:58.326061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.326054    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rpcn\" (UniqueName: \"kubernetes.io/projected/c402efd7-2d02-45e4-a94c-e65e09da4c39-kube-api-access-4rpcn\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:58.326061 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.326065    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c402efd7-2d02-45e4-a94c-e65e09da4c39-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:40:58.886564 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.886472    2562 generic.go:356] "Generic (PLEG): container finished" podID="c402efd7-2d02-45e4-a94c-e65e09da4c39" containerID="c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c" exitCode=0
Jul 14 19:40:58.886564 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.886536    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"
Jul 14 19:40:58.887018 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.886561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" event={"ID":"c402efd7-2d02-45e4-a94c-e65e09da4c39","Type":"ContainerDied","Data":"c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c"}
Jul 14 19:40:58.887018 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.886637    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv" event={"ID":"c402efd7-2d02-45e4-a94c-e65e09da4c39","Type":"ContainerDied","Data":"637f41edbde363dc0136f62fab642b7b5b40f369ef9e91ab2566740ab5b02eee"}
Jul 14 19:40:58.887018 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.886659    2562 scope.go:117] "RemoveContainer" containerID="c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c"
Jul 14 19:40:58.895799 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.895780    2562 scope.go:117] "RemoveContainer" containerID="c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c"
Jul 14 19:40:58.896067 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:40:58.896045    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c\": container with ID starting with c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c not found: ID does not exist" containerID="c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c"
Jul 14 19:40:58.896126 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.896080    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c"} err="failed to get container status \"c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c\": rpc error: code = NotFound desc = could not find container \"c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c\": container with ID starting with c5ac9c8bdff32d7f5668479a82a50190e1f13b526128d0e609b24ae68211b44c not found: ID does not exist"
Jul 14 19:40:58.918198 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.918171    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"]
Jul 14 19:40:58.926317 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:40:58.926292    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-755a0b-55dcdf8b5-dptrv"]
Jul 14 19:41:00.621545 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:00.621512    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c402efd7-2d02-45e4-a94c-e65e09da4c39" path="/var/lib/kubelet/pods/c402efd7-2d02-45e4-a94c-e65e09da4c39/volumes"
Jul 14 19:41:02.303749 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:02.303713    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:09.787560 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:09.787525    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:09.876380 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:09.876343    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:13.017556 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.017523    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:13.367527 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367446    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"]
Jul 14 19:41:13.367809 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c402efd7-2d02-45e4-a94c-e65e09da4c39" containerName="maas-api"
Jul 14 19:41:13.367853 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367809    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c402efd7-2d02-45e4-a94c-e65e09da4c39" containerName="maas-api"
Jul 14 19:41:13.367853 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367819    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ffe92b-96ad-4715-bc4d-e4058d33a69d" containerName="revoke-keys"
Jul 14 19:41:13.367853 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367826    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ffe92b-96ad-4715-bc4d-e4058d33a69d" containerName="revoke-keys"
Jul 14 19:41:13.367963 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367913    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c402efd7-2d02-45e4-a94c-e65e09da4c39" containerName="maas-api"
Jul 14 19:41:13.367963 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.367920    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ffe92b-96ad-4715-bc4d-e4058d33a69d" containerName="revoke-keys"
Jul 14 19:41:13.372321 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.372300    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:13.375329 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.375306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sgh6v\""
Jul 14 19:41:13.382961 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.382938    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"]
Jul 14 19:41:13.458356 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.458320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c69457-c94e-4b3f-b61d-b64e46109343-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz\" (UID: \"f1c69457-c94e-4b3f-b61d-b64e46109343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:13.559776 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.559735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c69457-c94e-4b3f-b61d-b64e46109343-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz\" (UID: \"f1c69457-c94e-4b3f-b61d-b64e46109343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:13.560346 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.560327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c69457-c94e-4b3f-b61d-b64e46109343-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz\" (UID: \"f1c69457-c94e-4b3f-b61d-b64e46109343\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:13.686066 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.685979    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:13.827578 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.827546    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"]
Jul 14 19:41:13.828526 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:41:13.828497    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c69457_c94e_4b3f_b61d_b64e46109343.slice/crio-ff551246db41bc5a8cbf5086cfdf28960ce3eb85f8cd135d3d821683a7d4ef72 WatchSource:0}: Error finding container ff551246db41bc5a8cbf5086cfdf28960ce3eb85f8cd135d3d821683a7d4ef72: Status 404 returned error can't find the container with id ff551246db41bc5a8cbf5086cfdf28960ce3eb85f8cd135d3d821683a7d4ef72
Jul 14 19:41:13.961740 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:13.961634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz" event={"ID":"f1c69457-c94e-4b3f-b61d-b64e46109343","Type":"ContainerStarted","Data":"ff551246db41bc5a8cbf5086cfdf28960ce3eb85f8cd135d3d821683a7d4ef72"}
Jul 14 19:41:14.525095 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:14.525060    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:14.621888 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:14.621864    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:14.968862 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:14.968827    2562 generic.go:356] "Generic (PLEG): container finished" podID="f1c69457-c94e-4b3f-b61d-b64e46109343" containerID="8af30ae23fb5a1fdfa064ae6ab1fec4c8751016f2f701d1d0e83608cceebe9f7" exitCode=0
Jul 14 19:41:14.969028 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:14.968907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz" event={"ID":"f1c69457-c94e-4b3f-b61d-b64e46109343","Type":"ContainerDied","Data":"8af30ae23fb5a1fdfa064ae6ab1fec4c8751016f2f701d1d0e83608cceebe9f7"}
Jul 14 19:41:16.107748 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.107723    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:16.183166 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.183135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c69457-c94e-4b3f-b61d-b64e46109343-maas-api-service-ca\") pod \"f1c69457-c94e-4b3f-b61d-b64e46109343\" (UID: \"f1c69457-c94e-4b3f-b61d-b64e46109343\") "
Jul 14 19:41:16.183576 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.183547    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c69457-c94e-4b3f-b61d-b64e46109343-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1c69457-c94e-4b3f-b61d-b64e46109343" (UID: "f1c69457-c94e-4b3f-b61d-b64e46109343"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:41:16.283893 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.283799    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c69457-c94e-4b3f-b61d-b64e46109343-maas-api-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:41:16.986436 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.986385    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz"
Jul 14 19:41:16.986576 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.986431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz" event={"ID":"f1c69457-c94e-4b3f-b61d-b64e46109343","Type":"ContainerDied","Data":"ff551246db41bc5a8cbf5086cfdf28960ce3eb85f8cd135d3d821683a7d4ef72"}
Jul 14 19:41:16.986576 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:16.986519    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff551246db41bc5a8cbf5086cfdf28960ce3eb85f8cd135d3d821683a7d4ef72"
Jul 14 19:41:24.052979 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.052946    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"]
Jul 14 19:41:24.053458 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.053203    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb" podUID="e1441875-274d-4cf6-a565-49e9df408627" containerName="maas-api" containerID="cri-o://5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb" gracePeriod=30
Jul 14 19:41:24.113107 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.113056    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:24.305127 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.305061    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:41:24.458416 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.458314    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1441875-274d-4cf6-a565-49e9df408627-maas-api-tls\") pod \"e1441875-274d-4cf6-a565-49e9df408627\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") "
Jul 14 19:41:24.458416 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.458353    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrcm6\" (UniqueName: \"kubernetes.io/projected/e1441875-274d-4cf6-a565-49e9df408627-kube-api-access-hrcm6\") pod \"e1441875-274d-4cf6-a565-49e9df408627\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") "
Jul 14 19:41:24.458416 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.458386    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1441875-274d-4cf6-a565-49e9df408627-openshift-service-ca\") pod \"e1441875-274d-4cf6-a565-49e9df408627\" (UID: \"e1441875-274d-4cf6-a565-49e9df408627\") "
Jul 14 19:41:24.458831 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.458804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1441875-274d-4cf6-a565-49e9df408627-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1441875-274d-4cf6-a565-49e9df408627" (UID: "e1441875-274d-4cf6-a565-49e9df408627"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 19:41:24.460745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.460716    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1441875-274d-4cf6-a565-49e9df408627-kube-api-access-hrcm6" (OuterVolumeSpecName: "kube-api-access-hrcm6") pod "e1441875-274d-4cf6-a565-49e9df408627" (UID: "e1441875-274d-4cf6-a565-49e9df408627"). InnerVolumeSpecName "kube-api-access-hrcm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 19:41:24.460852 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.460746    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1441875-274d-4cf6-a565-49e9df408627-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1441875-274d-4cf6-a565-49e9df408627" (UID: "e1441875-274d-4cf6-a565-49e9df408627"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 19:41:24.559220 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.559124    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1441875-274d-4cf6-a565-49e9df408627-openshift-service-ca\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:41:24.559220 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.559162    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1441875-274d-4cf6-a565-49e9df408627-maas-api-tls\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:41:24.559220 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:24.559175    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrcm6\" (UniqueName: \"kubernetes.io/projected/e1441875-274d-4cf6-a565-49e9df408627-kube-api-access-hrcm6\") on node \"ip-10-0-132-123.ec2.internal\" DevicePath \"\""
Jul 14 19:41:25.026890 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.026854    2562 generic.go:356] "Generic (PLEG): container finished" podID="e1441875-274d-4cf6-a565-49e9df408627" containerID="5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb" exitCode=0
Jul 14 19:41:25.027114 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.026923    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"
Jul 14 19:41:25.027114 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.026937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb" event={"ID":"e1441875-274d-4cf6-a565-49e9df408627","Type":"ContainerDied","Data":"5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb"}
Jul 14 19:41:25.027114 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.027029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb" event={"ID":"e1441875-274d-4cf6-a565-49e9df408627","Type":"ContainerDied","Data":"f6ec718711e823bfd7f84e84dd72cb9c80ff4c8a048bf59844d86a6a18b73be8"}
Jul 14 19:41:25.027114 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.027055    2562 scope.go:117] "RemoveContainer" containerID="5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb"
Jul 14 19:41:25.035971 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.035955    2562 scope.go:117] "RemoveContainer" containerID="5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb"
Jul 14 19:41:25.036226 ip-10-0-132-123 kubenswrapper[2562]: E0714 19:41:25.036206    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb\": container with ID starting with 5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb not found: ID does not exist" containerID="5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb"
Jul 14 19:41:25.036287 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.036233    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb"} err="failed to get container status \"5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb\": rpc error: code = NotFound desc = could not find container \"5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb\": container with ID starting with 5019fb434ae184fcb696ce5e330695b7c1861ee6c256c3da4e5dbe80afb364fb not found: ID does not exist"
Jul 14 19:41:25.045917 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.045893    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"]
Jul 14 19:41:25.050353 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:25.050330    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7c2fbe-7b464b97f7-75jxb"]
Jul 14 19:41:26.622019 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:26.621984    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1441875-274d-4cf6-a565-49e9df408627" path="/var/lib/kubelet/pods/e1441875-274d-4cf6-a565-49e9df408627/volumes"
Jul 14 19:41:28.494075 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:28.494045    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:35.220261 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:35.220226    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:35.322362 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:35.322332    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:39.188964 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:39.188927    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:39.404614 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:39.404565    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:39.815075 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:39.815044    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66bc479cdc-758wm_14f4db86-596c-4d18-83a7-7ca3ece00d77/authorino/0.log"
Jul 14 19:41:45.425591 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:45.425558    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tc4h8_35434038-8fd6-4337-ab44-5b975cba860a/manager/0.log"
Jul 14 19:41:46.127909 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:46.127882    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff67ccb56-w97qn_4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899/copy-manifests/0.log"
Jul 14 19:41:46.143380 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:46.143353    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff67ccb56-w97qn_4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899/manager/0.log"
Jul 14 19:41:46.770831 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:46.770801    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734275-srpzd_010cf5d0-ccea-416a-94d3-05ea84e1e95f/cleanup/0.log"
Jul 14 19:41:46.891271 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:46.891245    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734290-4w7vs_8b9ee742-414b-4738-ae13-8762e3fc225d/cleanup/0.log"
Jul 14 19:41:47.009177 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.009144    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-1877882c-k4spj_c8dfc0f5-d267-41e5-8808-df916e07a249/revoke-keys/0.log"
Jul 14 19:41:47.124642 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.124561    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-61c60be8-9rbmp_f7081b84-acd7-476e-8a0e-10582624f2b4/revoke-keys/0.log"
Jul 14 19:41:47.245700 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.245672    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-de8e7b-pj6r5_2c627736-fe42-4a52-ab27-fb9ad03e31a3/revoke-keys/0.log"
Jul 14 19:41:47.357367 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.357340    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-8a10a8-hrfr4_339b04d3-bb53-4daf-9748-4fca5bda59fe/revoke-keys/0.log"
Jul 14 19:41:47.469985 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.469957    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-4961650e-phfm7_b49eed0a-e2ec-44f5-8e4b-754e85a49c14/revoke-keys/0.log"
Jul 14 19:41:47.712802 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.712771    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-feadd1-tdn2k_cb2ae5f3-c73d-4e55-8807-82dad39c30a0/revoke-keys/0.log"
Jul 14 19:41:47.830011 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.829936    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-040d121d-wcxns_10b1716d-9c75-49f0-a858-137465299a0a/revoke-keys/0.log"
Jul 14 19:41:47.958508 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:47.958480    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-393716c6-skmsk_1c022d2f-a9eb-474e-a117-66fc66b8b126/revoke-keys/0.log"
Jul 14 19:41:48.074082 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:48.074050    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-618aef4e-2h5td_c78ecefb-8dbb-4229-9778-b42b817a6be9/revoke-keys/0.log"
Jul 14 19:41:48.186154 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:48.186067    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-755a0b-4jk7m_37ffe92b-96ad-4715-bc4d-e4058d33a69d/revoke-keys/0.log"
Jul 14 19:41:48.314313 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:48.314286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-7c2fbe-lbszz_f1c69457-c94e-4b3f-b61d-b64e46109343/revoke-keys/0.log"
Jul 14 19:41:49.786745 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:49.786712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66bc479cdc-758wm_14f4db86-596c-4d18-83a7-7ca3ece00d77/authorino/0.log"
Jul 14 19:41:51.466946 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:51.466911    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd89b4b77-n87wj_9f9818b6-905d-4c31-8558-f30e13e4aae4/kube-auth-proxy/0.log"
Jul 14 19:41:52.438999 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:52.438912    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-cctr8_34778d9e-c272-4861-8884-d06f291d0742/main/0.log"
Jul 14 19:41:52.559445 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:52.559419    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bdfff_86a50ecc-56e8-4ee1-b654-a6c4a7b5bbd4/main/0.log"
Jul 14 19:41:52.924335 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:52.924305    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xqtnw_47be0a3e-e410-4219-bd1f-9061d126eacb/main/0.log"
Jul 14 19:41:58.557249 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:58.557210    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:41:58.701742 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:41:58.701698    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:01.904361 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:01.904321    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ch6sn_139756f3-c948-4ecc-b8e1-a222e73af648/global-pull-secret-syncer/0.log"
Jul 14 19:42:01.969612 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:01.969580    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log"
Jul 14 19:42:02.123090 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:02.123001    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-123.ec2.internal_ad3d36c070043dd6db6d4a5194d07329/haproxy/0.log"
Jul 14 19:42:07.282214 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:07.282174    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66bc479cdc-758wm_14f4db86-596c-4d18-83a7-7ca3ece00d77/authorino/0.log"
Jul 14 19:42:07.808914 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:07.808873    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:09.782076 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:09.782043    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zdfnf_603ef2e5-892c-428c-87e0-c797daa806c9/kube-state-metrics/0.log"
Jul 14 19:42:09.819900 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:09.819861    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zdfnf_603ef2e5-892c-428c-87e0-c797daa806c9/kube-rbac-proxy-main/0.log"
Jul 14 19:42:09.849693 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:09.849597    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zdfnf_603ef2e5-892c-428c-87e0-c797daa806c9/kube-rbac-proxy-self/0.log"
Jul 14 19:42:10.050798 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.050768    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrpjf_a320ccfe-ab58-473e-ad7f-b38904672c73/node-exporter/0.log"
Jul 14 19:42:10.080141 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.080107    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrpjf_a320ccfe-ab58-473e-ad7f-b38904672c73/kube-rbac-proxy/0.log"
Jul 14 19:42:10.109237 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.109167    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrpjf_a320ccfe-ab58-473e-ad7f-b38904672c73/init-textfile/0.log"
Jul 14 19:42:10.247737 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.247702    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5lvz9_f1ccaaeb-5935-43d3-b92e-765e26cde0a1/kube-rbac-proxy-main/0.log"
Jul 14 19:42:10.278999 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.278964    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5lvz9_f1ccaaeb-5935-43d3-b92e-765e26cde0a1/kube-rbac-proxy-self/0.log"
Jul 14 19:42:10.305368 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.305333    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-5lvz9_f1ccaaeb-5935-43d3-b92e-765e26cde0a1/openshift-state-metrics/0.log"
Jul 14 19:42:10.351724 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.351692    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/prometheus/0.log"
Jul 14 19:42:10.378173 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.378091    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/config-reloader/0.log"
Jul 14 19:42:10.415898 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.415853    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/thanos-sidecar/0.log"
Jul 14 19:42:10.446085 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.446062    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/kube-rbac-proxy-web/0.log"
Jul 14 19:42:10.475895 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.475869    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/kube-rbac-proxy/0.log"
Jul 14 19:42:10.509837 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.509805    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/kube-rbac-proxy-thanos/0.log"
Jul 14 19:42:10.538215 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.538180    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f9e94bd1-5c75-4816-ae29-022f5b8152ad/init-config-reloader/0.log"
Jul 14 19:42:10.646470 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.646363    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-gqzm8_1f0190c6-079f-4e91-9018-168e539f5fdf/prometheus-operator-admission-webhook/0.log"
Jul 14 19:42:10.776874 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.776847    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c6f56d6c-mnc26_2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4/thanos-query/0.log"
Jul 14 19:42:10.810506 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.810479    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c6f56d6c-mnc26_2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4/kube-rbac-proxy-web/0.log"
Jul 14 19:42:10.837822 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.837789    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c6f56d6c-mnc26_2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4/kube-rbac-proxy/0.log"
Jul 14 19:42:10.869007 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.868981    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c6f56d6c-mnc26_2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4/prom-label-proxy/0.log"
Jul 14 19:42:10.899424 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.899315    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c6f56d6c-mnc26_2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4/kube-rbac-proxy-rules/0.log"
Jul 14 19:42:10.927604 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:10.927568    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c6f56d6c-mnc26_2bc64e0a-bde8-4094-a2b6-b0dd3ee74cd4/kube-rbac-proxy-metrics/0.log"
Jul 14 19:42:11.586789 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.586759    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"]
Jul 14 19:42:11.587119 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.587106    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1441875-274d-4cf6-a565-49e9df408627" containerName="maas-api"
Jul 14 19:42:11.587171 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.587121    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1441875-274d-4cf6-a565-49e9df408627" containerName="maas-api"
Jul 14 19:42:11.587171 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.587130    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c69457-c94e-4b3f-b61d-b64e46109343" containerName="revoke-keys"
Jul 14 19:42:11.587171 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.587136    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c69457-c94e-4b3f-b61d-b64e46109343" containerName="revoke-keys"
Jul 14 19:42:11.587283 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.587199    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1441875-274d-4cf6-a565-49e9df408627" containerName="maas-api"
Jul 14 19:42:11.587283 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.587207    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c69457-c94e-4b3f-b61d-b64e46109343" containerName="revoke-keys"
Jul 14 19:42:11.590337 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.590317    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.594062 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.594046    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pkg8j\"/\"kube-root-ca.crt\""
Jul 14 19:42:11.594818 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.594802    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pkg8j\"/\"default-dockercfg-bdtdn\""
Jul 14 19:42:11.595056 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.595041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pkg8j\"/\"openshift-service-ca.crt\""
Jul 14 19:42:11.605873 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.605852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"]
Jul 14 19:42:11.675177 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.675137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-proc\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.675360 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.675182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-lib-modules\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.675360 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.675212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-sys\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.675360 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.675288    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-podres\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.675501 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.675365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg89z\" (UniqueName: \"kubernetes.io/projected/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-kube-api-access-sg89z\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776166 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-proc\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776166 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-lib-modules\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-proc\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-lib-modules\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-sys\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776323    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-podres\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776367    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg89z\" (UniqueName: \"kubernetes.io/projected/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-kube-api-access-sg89z\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776429 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776382    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-sys\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.776625 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.776541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-podres\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.790439 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.790394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg89z\" (UniqueName: \"kubernetes.io/projected/dd594cf9-19f5-4698-9bbe-1b32b7e50c8e-kube-api-access-sg89z\") pod \"perf-node-gather-daemonset-j87ht\" (UID: \"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e\") " pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:11.899894 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:11.899788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:12.042882 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:12.042858    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"]
Jul 14 19:42:12.044779 ip-10-0-132-123 kubenswrapper[2562]: W0714 19:42:12.044747    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd594cf9_19f5_4698_9bbe_1b32b7e50c8e.slice/crio-a7b38b7670989918be63e46f3d14a789524249911c4eb11358d2a041c285d1c2 WatchSource:0}: Error finding container a7b38b7670989918be63e46f3d14a789524249911c4eb11358d2a041c285d1c2: Status 404 returned error can't find the container with id a7b38b7670989918be63e46f3d14a789524249911c4eb11358d2a041c285d1c2
Jul 14 19:42:12.273523 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:12.273483    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht" event={"ID":"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e","Type":"ContainerStarted","Data":"2d70cea82d865f93ffce11c5faa4cb6db5493958dfba964dbf4897c1c1587b49"}
Jul 14 19:42:12.273697 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:12.273543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht" event={"ID":"dd594cf9-19f5-4698-9bbe-1b32b7e50c8e","Type":"ContainerStarted","Data":"a7b38b7670989918be63e46f3d14a789524249911c4eb11358d2a041c285d1c2"}
Jul 14 19:42:12.273697 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:12.273571    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:12.295248 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:12.295200    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht" podStartSLOduration=1.2951781549999999 podStartE2EDuration="1.295178155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:42:12.293498392 +0000 UTC m=+2876.166056611" watchObservedRunningTime="2026-07-14 19:42:12.295178155 +0000 UTC m=+2876.167736375"
Jul 14 19:42:13.185447 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:13.185393    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:13.282383 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:13.282346    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:14.054238 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:14.054204    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:15.472692 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:15.472668    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-96h9g_e4877449-bd14-4f21-ae2c-4337ae659d5a/dns/0.log"
Jul 14 19:42:15.516419 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:15.516382    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-96h9g_e4877449-bd14-4f21-ae2c-4337ae659d5a/kube-rbac-proxy/0.log"
Jul 14 19:42:15.775879 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:15.775797    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwrp6_a3a0c652-ff2b-4ada-8de8-584ef8d6166b/dns-node-resolver/0.log"
Jul 14 19:42:16.476080 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:16.476044    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78b49755bc-gtvpd_33f1b5cf-6215-461b-a248-5707fd96cb1d/registry/0.log"
Jul 14 19:42:16.571796 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:16.571757    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xl9rj_918653c0-439e-4d9d-81ad-e777db1b9eab/node-ca/0.log"
Jul 14 19:42:17.992119 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:17.992090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd89b4b77-n87wj_9f9818b6-905d-4c31-8558-f30e13e4aae4/kube-auth-proxy/0.log"
Jul 14 19:42:18.289529 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:18.289457    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pkg8j/perf-node-gather-daemonset-j87ht"
Jul 14 19:42:18.427634 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:18.427603    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:18.501574 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:18.501537    2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2v9cv_e02c79d6-96ac-4231-a203-bcd7c713fa6c/konnectivity-agent/0.log" limit=1024
Jul 14 19:42:18.970045 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:18.970018    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jv45g_8bd5555d-a0a4-40a2-ad7d-83273d5fb2bf/serve-healthcheck-canary/0.log"
Jul 14 19:42:19.547243 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:19.547218    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96rqs_f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9/kube-rbac-proxy/0.log"
Jul 14 19:42:19.576507 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:19.576483    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96rqs_f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9/exporter/0.log"
Jul 14 19:42:19.603318 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:19.603292    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-96rqs_f3ce78ff-e3f1-4c2f-9566-3358fdf3fff9/extractor/0.log"
Jul 14 19:42:22.196764 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:22.196735    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd89b4b77-n87wj_9f9818b6-905d-4c31-8558-f30e13e4aae4/kube-auth-proxy/0.log"
Jul 14 19:42:22.425121 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:22.425084    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tc4h8_35434038-8fd6-4337-ab44-5b975cba860a/manager/0.log"
Jul 14 19:42:22.883620 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:22.883596    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff67ccb56-w97qn_4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899/manager/0.log"
Jul 14 19:42:22.953513 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:22.953490    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff67ccb56-w97qn_4c907d3f-ecd0-4fa1-80ea-9c0ed58e8899/copy-manifests/0.log"
Jul 14 19:42:23.845731 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:23.845694    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd89b4b77-n87wj_9f9818b6-905d-4c31-8558-f30e13e4aae4/kube-auth-proxy/0.log"
Jul 14 19:42:24.399880 ip-10-0-132-123 kubenswrapper[2562]: I0714 19:42:24.399850    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c74864f54-wndml_30604a39-56e0-44d2-97e1-34ab12984b23/manager/0.log"