Jul 07 13:16:32.842952 ip-10-0-133-157 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 13:16:33.328418 ip-10-0-133-157 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 13:16:33.328418 ip-10-0-133-157 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 13:16:33.328418 ip-10-0-133-157 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 13:16:33.328418 ip-10-0-133-157 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 13:16:33.328418 ip-10-0-133-157 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 13:16:33.330049 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.329968    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 13:16:33.334685 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334671    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 13:16:33.334685 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334685    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334688    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334691    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334694    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334698    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334701    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334704    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334706    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334709    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334711    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334715    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334717    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334719    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334723    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334725    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334728    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334730    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334733    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334744    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334748    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 13:16:33.334757 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334752    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334755    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334758    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334760    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334763    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334766    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334769    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334772    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334774    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334777    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334779    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334782    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334784    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334787    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334789    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334792    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334794    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334797    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334799    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334802    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 13:16:33.335279 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334804    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334807    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334809    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334811    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334814    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334816    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334819    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334823    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334826    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334829    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334832    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334834    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334837    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334839    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334843    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334846    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334848    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334852    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334856    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 13:16:33.335815 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334859    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334861    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334864    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334866    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334869    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334871    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334874    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334876    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334879    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334881    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334884    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334887    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334889    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334892    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334895    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334897    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334899    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334902    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334904    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 13:16:33.336335 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334908    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334910    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334912    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334915    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334917    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334920    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334922    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334925    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334928    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334931    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334933    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334936    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334938    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334941    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334943    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334946    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334948    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334950    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334952    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.334955    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 13:16:33.336843 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335611    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335617    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335620    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335622    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335625    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335644    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335647    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335650    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335652    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335655    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335658    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335661    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335664    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335667    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335669    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335673    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335677    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335680    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335682    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 13:16:33.337390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335686    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335690    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335692    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335695    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335699    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335702    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335705    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335708    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335711    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335714    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335716    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335719    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335722    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335725    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335727    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335730    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335732    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335735    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335737    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335741    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 13:16:33.337902 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335743    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335745    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335748    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335750    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335753    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335755    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335757    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335760    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335763    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335765    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335767    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335770    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335773    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335777    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335780    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335782    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335786    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335789    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335793    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 13:16:33.338461 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335795    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335798    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335801    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335803    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335806    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335808    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335810    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335813    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335815    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335818    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335821    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335823    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335825    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335829    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335831    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335834    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335836    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335838    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335841    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 13:16:33.338973 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335843    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335846    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335848    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335850    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335853    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335855    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335858    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335861    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335864    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335866    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335869    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335871    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335873    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335876    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335878    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335881    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335883    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335885    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335888    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335890    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335892    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 13:16:33.339488 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.335894    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336265    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336271    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336274    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336276    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336279    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336282    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336285    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336287    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336290    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336292    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336295    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336297    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336299    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336302    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336304    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336306    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336310    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336316    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 13:16:33.340051 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336318    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336321    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336324    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336327    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336329    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336332    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336334    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336337    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336340    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336342    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336344    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336347    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336349    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336352    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336356    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336359    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336362    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336364    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336367    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 13:16:33.340548 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336371    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336374    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336376    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336378    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336381    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336383    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336386    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336388    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336391    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336393    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336396    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336398    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336402    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336406    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336409    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336412    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336415    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336418    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336421    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336424    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 13:16:33.341139 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336427    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336429    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336432    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336434    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336436    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336439    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336441    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336444    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336446    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336449    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336451    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336454    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336456    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336459    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336461    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336463    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336466    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336468    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336471    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336473    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 13:16:33.341663 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336475    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336478    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336481    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336483    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336487    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336489    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336491    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336494    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336497    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336499    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336503    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336505    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336508    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336511    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336513    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336515    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336519    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336522    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336524    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 13:16:33.342174 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336526    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336529    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.336531    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336596    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336602    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336607    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336611    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336616    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336619    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336623    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336639    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336643    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336645    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336648    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336652    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336655    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336658    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336663    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336666    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336669    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336671    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336675    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336677    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 13:16:33.342665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336680    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336683    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336686    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336690    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336693    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336696    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336699    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336702    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336705    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336707    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336710    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336713    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336717    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336720    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336722    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336725    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336728    2581 flags.go:64] FLAG: --enable-server="true"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336731    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336736    2581 flags.go:64] FLAG: --event-burst="100"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336739    2581 flags.go:64] FLAG: --event-qps="50"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336742    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336745    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336747    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336751    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 13:16:33.343297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336754    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336757    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336761    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336763    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336766    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336769    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336772    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336774    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336777    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336780    2581 flags.go:64] FLAG: --feature-gates=""
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336783    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336786    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336789    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336792    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336795    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336798    2581 flags.go:64] FLAG: --help="false"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336800    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336803    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336806    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336809    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336812    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336815    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336818    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 13:16:33.343939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336821    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336823    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336826    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336829    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336832    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336834    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336837    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336839    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336842    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336845    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336848    2581 flags.go:64] FLAG: --lock-file=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336851    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336854    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336857    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336862    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336865    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336868    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336870    2581 flags.go:64] FLAG: --logging-format="text"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336873    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336876    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336878    2581 flags.go:64] FLAG: --manifest-url=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336881    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336885    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336888    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336891    2581 flags.go:64] FLAG: --max-pods="110"
Jul 07 13:16:33.344598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336894    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336897    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336899    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336902    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336905    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336907    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336910    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336917    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336920    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336923    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336925    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336928    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336933    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336936    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336938    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336941    2581 flags.go:64] FLAG: --port="10250"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336944    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336947    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02be3a3b80ddf78d1"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336950    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336953    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336956    2581 flags.go:64] FLAG: --register-node="true"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336959    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336963    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336965    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 07 13:16:33.345271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336968    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336971    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336974    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336977    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336979    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336982    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336985    2581 flags.go:64] FLAG: --runonce="false"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336987    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336990    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336993    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336995    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.336998    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337001    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337004    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337007    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337009    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337012    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337014    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337017    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337020    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337023    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337026    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337030    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337033    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337037    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 13:16:33.345918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337040    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337043    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337046    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337049    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337052    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337056    2581 flags.go:64] FLAG: --v="2"
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337059    2581 flags.go:64] FLAG: --version="false"
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337063    2581 flags.go:64] FLAG: --vmodule=""
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337066    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.337070    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337194    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337197    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337200    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337203    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337206    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337210    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337212    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337215    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337218    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337221    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337223    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337226    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 13:16:33.346562 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337228    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337231    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337234    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337236    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337239    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337242    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337246    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337248    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337251    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337255    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337257    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337260    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337262    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337264    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337267    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337269    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337271    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337274    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337276    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337279    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 13:16:33.347157 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337281    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337284    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337286    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337288    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337291    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337294    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337296    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337299    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337301    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337304    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337306    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337309    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337311    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337313    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337316    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337318    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337321    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337323    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337325    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337328    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 13:16:33.347717 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337332    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337337    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337340    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337344    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337347    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337349    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337352    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337354    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337357    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337359    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337362    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337364    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337367    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337370    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337372    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337374    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337377    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337379    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337382    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 13:16:33.348276 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337384    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337387    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337389    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337391    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337394    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337397    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337399    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337403    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337406    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337409    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337412    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337415    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337418    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337421    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337426    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337430    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337435    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337438    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337440    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 13:16:33.348854 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337443    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337445    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337447    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337450    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337452    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337455    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337458    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337460    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.337462    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.338253    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.344807    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.344819    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.344837    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.344843    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344909    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 13:16:33.349446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344914    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344917    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344920    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344923    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344926    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344929    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344932    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344935    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344937    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344940    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344942    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344944    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344947    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344949    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344953    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344955    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344958    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344960    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344962    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344965    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 13:16:33.349863 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344968    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344970    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344973    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344975    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344978    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344981    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344984    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344986    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344989    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344992    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344994    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.344997    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345000    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345002    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345005    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345007    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345010    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345012    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345015    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345017    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345019    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 13:16:33.350374 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345022    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345026    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345030    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345033    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345036    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345039    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345042    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345045    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345048    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345050    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345053    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345056    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345058    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345061    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345063    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345066    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345069    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345071    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345074    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 13:16:33.350938 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345077    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345080    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345082    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345085    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345087    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345090    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345092    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345095    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345098    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345101    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345104    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345107    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345109    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345112    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345114    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345117    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345119    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345122    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345124    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 13:16:33.351446 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345126    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345129    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345131    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345134    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345136    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345138    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345141    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345143    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345145    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345148    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345150    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345153    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345157    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345161    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345164    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345167    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345169    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345172    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 13:16:33.352065 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345175    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.345180    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345288    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345293    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345296    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345298    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345301    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345304    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345306    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345309    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345311    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345314    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345316    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345319    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345322    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 13:16:33.352521 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345324    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345326    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345329    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345331    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345333    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345335    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345338    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345340    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345343    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345345    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345347    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345350    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345353    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345355    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345358    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345361    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345363    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345366    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345368    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345371    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 13:16:33.352933 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345373    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345377    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345381    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345383    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345386    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345388    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345390    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345393    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345395    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345398    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345400    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345402    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345405    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345407    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345409    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345413    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345416    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345419    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345422    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 13:16:33.353439 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345424    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345428    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345430    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345433    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345436    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345438    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345441    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345444    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345447    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345450    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345453    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345456    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345458    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345460    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345463    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345465    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345467    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345470    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345472    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345475    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 13:16:33.353928 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345477    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345480    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345482    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345485    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345487    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345489    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345492    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345494    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345496    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345499    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345501    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345505    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345508    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345511    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345513    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345516    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345518    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345521    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345523    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345526    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 13:16:33.354443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345528    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345531    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345533    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345536    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345538    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345540    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:33.345543    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.345547    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.345671    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.348862    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.350186    2581 server.go:1019] "Starting client certificate rotation"
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.350285    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 13:16:33.354963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.350325    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 13:16:33.357565 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.357545    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 13:16:33.360570 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.360545    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 13:16:33.375103 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.375082    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 07 13:16:33.376971 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.376956    2581 log.go:25] "Validated CRI v1 image API"
Jul 07 13:16:33.377719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.377705    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 13:16:33.380181 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.380159    2581 fs.go:135] Filesystem UUIDs: map[36417808-4afc-40fd-8b32-5e93539d83ec:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7f27deab-6c9f-43cd-b2dc-441493e26676:/dev/nvme0n1p4]
Jul 07 13:16:33.380254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.380179    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 13:16:33.382152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.382132    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 13:16:33.386417 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.386305    2581 manager.go:216] Machine: {Timestamp:2026-07-07 13:16:33.384949442 +0000 UTC m=+0.408850527 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101018 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec3c7d10441517fd6927bbf383346 SystemUUID:ec2ec3c7-d104-4151-7fd6-927bbf383346 BootID:62cf0ab3-ef99-4a0c-a1e0-6491a346a8fb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:df:5c:cd:e8:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:df:5c:cd:e8:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:88:41:7a:a3:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 13:16:33.386417 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.386409    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 13:16:33.386546 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.386507    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 13:16:33.389825 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.389801    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 13:16:33.389963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.389829    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 13:16:33.390457 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.390448    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 13:16:33.390487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.390460    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 13:16:33.390487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.390467    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 13:16:33.391267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.391256    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 13:16:33.392396 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.392386    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 13:16:33.395854 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.395843    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 13:16:33.395959 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.395950    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 13:16:33.395993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.395980    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 13:16:33.395993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.395990    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 13:16:33.396048 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.396003    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 13:16:33.396048 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.396012    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 07 13:16:33.396048 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.396020    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 13:16:33.396344 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.396332    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 13:16:33.396384 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.396354    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 13:16:33.398349 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.398333    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 13:16:33.401187 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.400601    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 13:16:33.401281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.401215    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 13:16:33.402440 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402412    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 13:16:33.402440 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402444    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402454    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402472    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402486    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402494    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402506    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402515    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402527    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402537    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402550    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 13:16:33.402566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.402564    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 13:16:33.403347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.403326    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 13:16:33.403435 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.403360    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 13:16:33.405172 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.405147    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 13:16:33.405274 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.405155    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 13:16:33.406823 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.406804    2581 server.go:1268] "Started kubelet"
Jul 07 13:16:33.406914 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.406868    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 13:16:33.406971 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.406890    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 13:16:33.406971 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.406959    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 13:16:33.407720 ip-10-0-133-157 systemd[1]: Started Kubernetes Kubelet.
Jul 07 13:16:33.408351 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.408174    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 13:16:33.408645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.408616    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 13:16:33.410554 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.410537    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 13:16:33.411288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.411269    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5rf5m"
Jul 07 13:16:33.411805 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.411786    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 13:16:33.414419 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.413230    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-157.ec2.internal.18c0044fd7b78811  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-157.ec2.internal,UID:ip-10-0-133-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-157.ec2.internal,},FirstTimestamp:2026-07-07 13:16:33.406773265 +0000 UTC m=+0.430674329,LastTimestamp:2026-07-07 13:16:33.406773265 +0000 UTC m=+0.430674329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-157.ec2.internal,}"
Jul 07 13:16:33.415318 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.415300    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 13:16:33.415318 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.415305    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 13:16:33.415473 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.415455    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.415910 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.415853    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 13:16:33.415910 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.415900    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 13:16:33.415910 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.415914    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 13:16:33.416061 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416009    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 13:16:33.416061 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416019    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 13:16:33.416449 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416429    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5rf5m"
Jul 07 13:16:33.416678 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416659    2581 factory.go:55] Registering systemd factory
Jul 07 13:16:33.416678 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416681    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 07 13:16:33.416923 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416906    2581 factory.go:153] Registering CRI-O factory
Jul 07 13:16:33.416923 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416924    2581 factory.go:223] Registration of the crio container factory successfully
Jul 07 13:16:33.417070 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416971    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 13:16:33.417070 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.416994    2581 factory.go:103] Registering Raw factory
Jul 07 13:16:33.417070 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.417007    2581 manager.go:1196] Started watching for new ooms in manager
Jul 07 13:16:33.417543 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.417529    2581 manager.go:349] Starting recovery of all containers
Jul 07 13:16:33.419179 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.419155    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 13:16:33.419309 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.419284    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 13:16:33.429862 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.429714    2581 manager.go:354] Recovery completed
Jul 07 13:16:33.434992 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.434980    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.437678 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.437662    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.437760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.437690    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.437760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.437699    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.438242 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.438228    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 13:16:33.438242 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.438240    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 13:16:33.438242 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.438243    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.438359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.438255    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 13:16:33.439209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.439190    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.439299 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.439219    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.439299 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.439241    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.439299 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.439226    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-157.ec2.internal.18c0044fd98f17cf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-157.ec2.internal,UID:ip-10-0-133-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-157.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-157.ec2.internal,},FirstTimestamp:2026-07-07 13:16:33.437677519 +0000 UTC m=+0.461578584,LastTimestamp:2026-07-07 13:16:33.437677519 +0000 UTC m=+0.461578584,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-157.ec2.internal,}"
Jul 07 13:16:33.439459 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.439448    2581 policy_none.go:49] "None policy: Start"
Jul 07 13:16:33.439505 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.439464    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 13:16:33.439505 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.439490    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 13:16:33.440425 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.440415    2581 policy_none.go:47] "Start"
Jul 07 13:16:33.476150 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476132    2581 manager.go:337] "Starting Device Plugin manager"
Jul 07 13:16:33.476260 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.476159    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 13:16:33.476260 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476168    2581 server.go:87] "Starting device plugin registration server"
Jul 07 13:16:33.476407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476394    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 13:16:33.476460 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476410    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 13:16:33.476523 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476510    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 13:16:33.476606 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476592    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 13:16:33.476669 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.476608    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 13:16:33.477055 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.477038    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 13:16:33.477120 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.477083    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.577199 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.577175    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.578227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.578109    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.578227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.578134    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.578227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.578144    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.578227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.578168    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.584652 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.584603    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.584652 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.584623    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-157.ec2.internal\": node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.595042 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.595028    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.608912 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.608886    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 13:16:33.610239 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.610221    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 13:16:33.610312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.610253    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 13:16:33.610312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.610275    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 13:16:33.610312 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.610305    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 13:16:33.612552 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.612534    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 13:16:33.695566 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.695548    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.710707 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.710689    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"]
Jul 07 13:16:33.710758 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.710738    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.711583 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.711570    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.711659 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.711600    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.711659 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.711609    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.712993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.712982    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.713092 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.713139 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713103    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.713615 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713602    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.713703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713621    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.713703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713646    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.713703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713683    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.713807 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713706    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.713807 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.713718    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.715011 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.714998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.715056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.715024    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 13:16:33.715650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.715621    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 13:16:33.715701 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.715664    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 13:16:33.715701 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.715675    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeHasSufficientPID"
Jul 07 13:16:33.718306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.718289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8cad6aa2145f5ea3bbbbae2bfaaf983-config\") pod \"kube-apiserver-proxy-ip-10-0-133-157.ec2.internal\" (UID: \"a8cad6aa2145f5ea3bbbbae2bfaaf983\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.718350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.718316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2a914bd14bade07bf5d2dc3674ec896-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal\" (UID: \"f2a914bd14bade07bf5d2dc3674ec896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.718350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.718335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a914bd14bade07bf5d2dc3674ec896-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal\" (UID: \"f2a914bd14bade07bf5d2dc3674ec896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.734163 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.734145    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-157.ec2.internal\" not found" node="ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.737493 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.737477    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-157.ec2.internal\" not found" node="ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.796196 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.796176    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.818908 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.818888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8cad6aa2145f5ea3bbbbae2bfaaf983-config\") pod \"kube-apiserver-proxy-ip-10-0-133-157.ec2.internal\" (UID: \"a8cad6aa2145f5ea3bbbbae2bfaaf983\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.818982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.818910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2a914bd14bade07bf5d2dc3674ec896-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal\" (UID: \"f2a914bd14bade07bf5d2dc3674ec896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.818982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.818928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a914bd14bade07bf5d2dc3674ec896-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal\" (UID: \"f2a914bd14bade07bf5d2dc3674ec896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.818982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.818952    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2a914bd14bade07bf5d2dc3674ec896-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal\" (UID: \"f2a914bd14bade07bf5d2dc3674ec896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.818982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.818970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f2a914bd14bade07bf5d2dc3674ec896-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal\" (UID: \"f2a914bd14bade07bf5d2dc3674ec896\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.819104 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:33.818982    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8cad6aa2145f5ea3bbbbae2bfaaf983-config\") pod \"kube-apiserver-proxy-ip-10-0-133-157.ec2.internal\" (UID: \"a8cad6aa2145f5ea3bbbbae2bfaaf983\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:33.897197 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.897151    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:33.997665 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:33.997623    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:34.037159 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:34.037133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:34.041280 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:34.041011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:34.098342 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:34.098320    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:35.342367 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.342317    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:35.442947 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.442924    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-157.ec2.internal\" not found"
Jul 07 13:16:35.443403 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.443389    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 13:16:35.492664 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.492646    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 13:16:35.492664 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.492661    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 13:16:35.493174 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.492745    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 13:16:35.493174 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.492767    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 13:16:35.539680 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.539661    2581 apiserver.go:52] "Watching apiserver"
Jul 07 13:16:35.550667 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.550650    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 13:16:35.550987 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.550970    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-qdpl2","openshift-dns/node-resolver-c48gn","openshift-multus/multus-4srzl","openshift-network-operator/iptables-alerter-tvsmw","openshift-ovn-kubernetes/ovnkube-node-2pg9k","kube-system/global-pull-secret-syncer-mbpsm","kube-system/konnectivity-agent-bdq6p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7","openshift-image-registry/node-ca-ksnhj","openshift-multus/multus-additional-cni-plugins-z9bz7","openshift-multus/network-metrics-daemon-s6p2q","openshift-network-diagnostics/network-check-target-lxbcn"]
Jul 07 13:16:35.553488 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.553198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.554229 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.554209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.555746 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.555730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-svtlw\""
Jul 07 13:16:35.555878 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.555862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.556045 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.556032    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.556468 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.556455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-srdcm\""
Jul 07 13:16:35.556771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.556759    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.557097 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.557083    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.557850 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.557834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.559141 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.559122    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 13:16:35.559231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.559142    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:35.559271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.559241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.560056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.560041    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 13:16:35.560056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.560050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.560231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.560092    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.560231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.560127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hbkg\""
Jul 07 13:16:35.560231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.560046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 13:16:35.560883 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.560871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.562087 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.562074    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.562148 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.562130    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:35.563046 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.563031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 13:16:35.563236 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.563217    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76m5v\""
Jul 07 13:16:35.563358 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.563344    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.563473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.563460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.564015 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.564071 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564029    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d7vmg\""
Jul 07 13:16:35.564121 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 13:16:35.564121 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 13:16:35.564239 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564148    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.564239 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.564432 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564419    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 13:16:35.564522 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.564506    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 13:16:35.565035 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-modprobe-d\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565075 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-var-lib-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.565075 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-ovn\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.565137 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysctl-conf\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565137 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-lib-modules\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565137 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/635424cd-2fa4-482b-bfce-2952febc14e3-tmp\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-cnibin\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.565254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-daemon-config\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.565254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-slash\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.565254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-var-lib-kubelet\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/635424cd-2fa4-482b-bfce-2952febc14e3-etc-tuned\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-socket-dir-parent\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-systemd\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-cni-multus\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-hostroot\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.565402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66b21091-8600-485e-86d6-2b0293bc48ae-host-slash\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-sys\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4845995-0d75-4296-ad03-aea66ec32c37-hosts-file\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-k8s-cni-cncf-io\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-cni-netd\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-kubernetes\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4845995-0d75-4296-ad03-aea66ec32c37-tmp-dir\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66b21091-8600-485e-86d6-2b0293bc48ae-iptables-alerter-script\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-etc-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 13:16:35.565702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cd2k\" (UniqueName: \"kubernetes.io/projected/b60e128f-b64e-4026-b937-2f7d61b35524-kube-api-access-7cd2k\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-cni-binary-copy\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-kubelet\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5hgp5\""
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-etc-kubernetes\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-env-overrides\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-multus-certs\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysctl-d\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-run\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-kubelet\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-systemd-units\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-ovnkube-config\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.565994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysconfig\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-systemd\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-os-release\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-host\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-cni-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-log-socket\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psj7w\" (UniqueName: \"kubernetes.io/projected/635424cd-2fa4-482b-bfce-2952febc14e3-kube-api-access-psj7w\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-netns\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-run-netns\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-cni-bin\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-system-cni-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-conf-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj96c\" (UniqueName: \"kubernetes.io/projected/a4845995-0d75-4296-ad03-aea66ec32c37-kube-api-access-wj96c\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-cni-bin\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48vx\" (UniqueName: \"kubernetes.io/projected/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-kube-api-access-b48vx\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktdl\" (UniqueName: \"kubernetes.io/projected/66b21091-8600-485e-86d6-2b0293bc48ae-kube-api-access-tktdl\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-node-log\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-ovnkube-script-lib\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.566702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b60e128f-b64e-4026-b937-2f7d61b35524-ovn-node-metrics-cert\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.567272 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.566904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.567472 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.567458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.567528 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.567512    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 13:16:35.567620 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.567605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.567700 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.567682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bvqnd\""
Jul 07 13:16:35.568479 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.568465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.569050 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.569030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 13:16:35.569185 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.569095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zm4fg\""
Jul 07 13:16:35.569185 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.569118    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 13:16:35.569852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.569836    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 13:16:35.569932 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.569863    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 13:16:35.569932 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.569923    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"
Jul 07 13:16:35.570111 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.570099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:35.570189 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.570155    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:35.570926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.570907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 13:16:35.571018 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.570939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 13:16:35.571071 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.571027    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-slpd7\""
Jul 07 13:16:35.571675 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.571635    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal"]
Jul 07 13:16:35.571778 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.571765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:35.571881 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.571843    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:35.572667 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.572625    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 13:11:34 +0000 UTC" deadline="2028-02-02 23:10:40.458101127 +0000 UTC"
Jul 07 13:16:35.572667 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.572663    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13809h54m4.885440893s"
Jul 07 13:16:35.573596 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.573582    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 13:16:35.579963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.579947    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 13:16:35.580043 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.580016    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal"]
Jul 07 13:16:35.593976 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.593938    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ql5jd"
Jul 07 13:16:35.601273 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.601258    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ql5jd"
Jul 07 13:16:35.660713 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.660692    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 13:16:35.666814 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4845995-0d75-4296-ad03-aea66ec32c37-tmp-dir\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.666927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66b21091-8600-485e-86d6-2b0293bc48ae-iptables-alerter-script\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.666927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-etc-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.666927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cd2k\" (UniqueName: \"kubernetes.io/projected/b60e128f-b64e-4026-b937-2f7d61b35524-kube-api-access-7cd2k\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.666927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.666927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a209626-ef8e-403b-b8e4-4778a11c9e2d-konnectivity-ca\") pod \"konnectivity-agent-bdq6p\" (UID: \"2a209626-ef8e-403b-b8e4-4778a11c9e2d\") " pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.667128 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.666951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-cni-binary-copy\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667238 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4845995-0d75-4296-ad03-aea66ec32c37-tmp-dir\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.667238 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667204    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-etc-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-kubelet\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-etc-kubernetes\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667311    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-kubelet\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-env-overrides\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-os-release\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667359    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-etc-kubernetes\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-multus-certs\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667419    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66b21091-8600-485e-86d6-2b0293bc48ae-iptables-alerter-script\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b20eff8d-cbbe-4869-b380-13a25e5fabd0-serviceca\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667418    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-cni-binary-copy\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667447    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-multus-certs\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysctl-d\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-run\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-kubelet\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-systemd-units\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-ovnkube-config\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-cnibin\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667545    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysctl-d\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667562    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-kubelet\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667565    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-systemd-units\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667590    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-run\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.667719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysconfig\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-systemd\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81f506f0-9be6-48c3-b771-9e7648ada7a4-dbus\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667783    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-env-overrides\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667804    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-systemd\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667807    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysconfig\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbc6f\" (UniqueName: \"kubernetes.io/projected/7e9315e0-f682-4632-acf8-0574443059e8-kube-api-access-pbc6f\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-os-release\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-host\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667925    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-os-release\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-cni-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667958    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-host\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-log-socket\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667978    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-ovnkube-config\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.668591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-cni-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.667999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-registration-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668036    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-log-socket\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psj7w\" (UniqueName: \"kubernetes.io/projected/635424cd-2fa4-482b-bfce-2952febc14e3-kube-api-access-psj7w\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-netns\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-run-netns\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-cni-bin\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668139    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-netns\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81f506f0-9be6-48c3-b771-9e7648ada7a4-kubelet-config\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-cni-bin\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668211    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-run-netns\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-socket-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-device-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-system-cni-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-conf-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668336    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-system-cni-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.669430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668361    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-conf-dir\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-sys-fs\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj96c\" (UniqueName: \"kubernetes.io/projected/a4845995-0d75-4296-ad03-aea66ec32c37-kube-api-access-wj96c\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-cni-bin\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48vx\" (UniqueName: \"kubernetes.io/projected/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-kube-api-access-b48vx\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktdl\" (UniqueName: \"kubernetes.io/projected/66b21091-8600-485e-86d6-2b0293bc48ae-kube-api-access-tktdl\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-node-log\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668564    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-cni-bin\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-ovnkube-script-lib\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668611    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668632    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-node-log\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2tq\" (UniqueName: \"kubernetes.io/projected/b20eff8d-cbbe-4869-b380-13a25e5fabd0-kube-api-access-dg2tq\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b60e128f-b64e-4026-b937-2f7d61b35524-ovn-node-metrics-cert\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-modprobe-d\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-run-ovn-kubernetes\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-var-lib-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-ovn\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668861    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-modprobe-d\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysctl-conf\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668892    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-var-lib-openvswitch\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668902    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-run-ovn\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-lib-modules\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/635424cd-2fa4-482b-bfce-2952febc14e3-tmp\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-cnibin\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.668996    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-lib-modules\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-daemon-config\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669013    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-sysctl-conf\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-slash\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669065    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-slash\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.670708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-var-lib-kubelet\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/635424cd-2fa4-482b-bfce-2952febc14e3-etc-tuned\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-var-lib-kubelet\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669152    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-socket-dir-parent\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669231    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-cnibin\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669275    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-socket-dir-parent\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669292    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b60e128f-b64e-4026-b937-2f7d61b35524-ovnkube-script-lib\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b20eff8d-cbbe-4869-b380-13a25e5fabd0-host\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-systemd\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-cni-multus\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-hostroot\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66b21091-8600-485e-86d6-2b0293bc48ae-host-slash\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdgg7\" (UniqueName: \"kubernetes.io/projected/ac1483d8-866d-4145-a97a-ec743ae7395d-kube-api-access-cdgg7\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669440    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-var-lib-cni-multus\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669465    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-hostroot\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.671281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktc5\" (UniqueName: \"kubernetes.io/projected/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-kube-api-access-lktc5\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-sys\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4845995-0d75-4296-ad03-aea66ec32c37-hosts-file\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669559    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66b21091-8600-485e-86d6-2b0293bc48ae-host-slash\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-k8s-cni-cncf-io\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-cni-netd\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669591    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-multus-daemon-config\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a209626-ef8e-403b-b8e4-4778a11c9e2d-agent-certs\") pod \"konnectivity-agent-bdq6p\" (UID: \"2a209626-ef8e-403b-b8e4-4778a11c9e2d\") " pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669612    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-systemd\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669639    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-sys\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-kubernetes\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669670    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-host-run-k8s-cni-cncf-io\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669671    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b60e128f-b64e-4026-b937-2f7d61b35524-host-cni-netd\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669715    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4845995-0d75-4296-ad03-aea66ec32c37-hosts-file\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.672055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.669756    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635424cd-2fa4-482b-bfce-2952febc14e3-etc-kubernetes\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.672871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.672839    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/635424cd-2fa4-482b-bfce-2952febc14e3-tmp\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.673025 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.673007    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/635424cd-2fa4-482b-bfce-2952febc14e3-etc-tuned\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.673523 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.673503    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b60e128f-b64e-4026-b937-2f7d61b35524-ovn-node-metrics-cert\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.674945 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.674918    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cd2k\" (UniqueName: \"kubernetes.io/projected/b60e128f-b64e-4026-b937-2f7d61b35524-kube-api-access-7cd2k\") pod \"ovnkube-node-2pg9k\" (UID: \"b60e128f-b64e-4026-b937-2f7d61b35524\") " pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.675848 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.675830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psj7w\" (UniqueName: \"kubernetes.io/projected/635424cd-2fa4-482b-bfce-2952febc14e3-kube-api-access-psj7w\") pod \"tuned-qdpl2\" (UID: \"635424cd-2fa4-482b-bfce-2952febc14e3\") " pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.676318 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.676297    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj96c\" (UniqueName: \"kubernetes.io/projected/a4845995-0d75-4296-ad03-aea66ec32c37-kube-api-access-wj96c\") pod \"node-resolver-c48gn\" (UID: \"a4845995-0d75-4296-ad03-aea66ec32c37\") " pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.676431 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.676415    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktdl\" (UniqueName: \"kubernetes.io/projected/66b21091-8600-485e-86d6-2b0293bc48ae-kube-api-access-tktdl\") pod \"iptables-alerter-tvsmw\" (UID: \"66b21091-8600-485e-86d6-2b0293bc48ae\") " pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.676563 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.676538    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48vx\" (UniqueName: \"kubernetes.io/projected/4a5d83cc-f411-41ea-b874-b2f5aeb7426c-kube-api-access-b48vx\") pod \"multus-4srzl\" (UID: \"4a5d83cc-f411-41ea-b874-b2f5aeb7426c\") " pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.758216 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.758185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cad6aa2145f5ea3bbbbae2bfaaf983.slice/crio-fb8622be14c2ef5452110ab247ded019d4a5e4c9133b1118ddae824d4d84e1cc WatchSource:0}: Error finding container fb8622be14c2ef5452110ab247ded019d4a5e4c9133b1118ddae824d4d84e1cc: Status 404 returned error can't find the container with id fb8622be14c2ef5452110ab247ded019d4a5e4c9133b1118ddae824d4d84e1cc
Jul 07 13:16:35.758502 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.758482    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a914bd14bade07bf5d2dc3674ec896.slice/crio-5bd861d6f84af0e3450388ac002d9ce6dac0eeb5d5aae16fd2fac8a04ba52dc6 WatchSource:0}: Error finding container 5bd861d6f84af0e3450388ac002d9ce6dac0eeb5d5aae16fd2fac8a04ba52dc6: Status 404 returned error can't find the container with id 5bd861d6f84af0e3450388ac002d9ce6dac0eeb5d5aae16fd2fac8a04ba52dc6
Jul 07 13:16:35.762912 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.762898    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 13:16:35.770743 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.770818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.770818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b20eff8d-cbbe-4869-b380-13a25e5fabd0-host\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.770818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdgg7\" (UniqueName: \"kubernetes.io/projected/ac1483d8-866d-4145-a97a-ec743ae7395d-kube-api-access-cdgg7\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.770818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktc5\" (UniqueName: \"kubernetes.io/projected/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-kube-api-access-lktc5\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a209626-ef8e-403b-b8e4-4778a11c9e2d-agent-certs\") pod \"konnectivity-agent-bdq6p\" (UID: \"2a209626-ef8e-403b-b8e4-4778a11c9e2d\") " pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b20eff8d-cbbe-4869-b380-13a25e5fabd0-host\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770841    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-system-cni-dir\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770838    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a209626-ef8e-403b-b8e4-4778a11c9e2d-konnectivity-ca\") pod \"konnectivity-agent-bdq6p\" (UID: \"2a209626-ef8e-403b-b8e4-4778a11c9e2d\") " pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.770951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-os-release\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.770956    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.770981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b20eff8d-cbbe-4869-b380-13a25e5fabd0-serviceca\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771029    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-os-release\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.771041    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:36.27099733 +0000 UTC m=+2.151368602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-cnibin\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771144    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-cnibin\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81f506f0-9be6-48c3-b771-9e7648ada7a4-dbus\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbc6f\" (UniqueName: \"kubernetes.io/projected/7e9315e0-f682-4632-acf8-0574443059e8-kube-api-access-pbc6f\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771257    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81f506f0-9be6-48c3-b771-9e7648ada7a4-dbus\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-registration-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.771306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81f506f0-9be6-48c3-b771-9e7648ada7a4-kubelet-config\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-socket-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-device-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771402    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e9315e0-f682-4632-acf8-0574443059e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-sys-fs\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2tq\" (UniqueName: \"kubernetes.io/projected/b20eff8d-cbbe-4869-b380-13a25e5fabd0-kube-api-access-dg2tq\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771539    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771549    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771593    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a209626-ef8e-403b-b8e4-4778a11c9e2d-konnectivity-ca\") pod \"konnectivity-agent-bdq6p\" (UID: \"2a209626-ef8e-403b-b8e4-4778a11c9e2d\") " pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771668    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-device-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771673    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81f506f0-9be6-48c3-b771-9e7648ada7a4-kubelet-config\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771721    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-socket-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771735    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-registration-dir\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771671    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-sys-fs\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.771797    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:35.772113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771863    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e9315e0-f682-4632-acf8-0574443059e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.772647 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771869    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b20eff8d-cbbe-4869-b380-13a25e5fabd0-serviceca\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.772647 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.771880    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac1483d8-866d-4145-a97a-ec743ae7395d-etc-selinux\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.772647 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.771902    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:36.271886606 +0000 UTC m=+2.152257878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:35.773207 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.773192    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a209626-ef8e-403b-b8e4-4778a11c9e2d-agent-certs\") pod \"konnectivity-agent-bdq6p\" (UID: \"2a209626-ef8e-403b-b8e4-4778a11c9e2d\") " pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.776902 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.776882    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:16:35.777010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.776905    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:16:35.777010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.776919    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:35.777010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:35.776976    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:16:36.27696031 +0000 UTC m=+2.157331581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:35.778501 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.778482    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdgg7\" (UniqueName: \"kubernetes.io/projected/ac1483d8-866d-4145-a97a-ec743ae7395d-kube-api-access-cdgg7\") pod \"aws-ebs-csi-driver-node-sg4f7\" (UID: \"ac1483d8-866d-4145-a97a-ec743ae7395d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.778582 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.778546    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktc5\" (UniqueName: \"kubernetes.io/projected/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-kube-api-access-lktc5\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:35.779197 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.779179    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2tq\" (UniqueName: \"kubernetes.io/projected/b20eff8d-cbbe-4869-b380-13a25e5fabd0-kube-api-access-dg2tq\") pod \"node-ca-ksnhj\" (UID: \"b20eff8d-cbbe-4869-b380-13a25e5fabd0\") " pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.779263 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.779207    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbc6f\" (UniqueName: \"kubernetes.io/projected/7e9315e0-f682-4632-acf8-0574443059e8-kube-api-access-pbc6f\") pod \"multus-additional-cni-plugins-z9bz7\" (UID: \"7e9315e0-f682-4632-acf8-0574443059e8\") " pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:35.876824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.876768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qdpl2"
Jul 07 13:16:35.884699 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.884680    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635424cd_2fa4_482b_bfce_2952febc14e3.slice/crio-9ba59b6cc80cb366e6d11456e19cbc03cf9881a8d863e61376a738212b1d9c7b WatchSource:0}: Error finding container 9ba59b6cc80cb366e6d11456e19cbc03cf9881a8d863e61376a738212b1d9c7b: Status 404 returned error can't find the container with id 9ba59b6cc80cb366e6d11456e19cbc03cf9881a8d863e61376a738212b1d9c7b
Jul 07 13:16:35.890747 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.890733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c48gn"
Jul 07 13:16:35.897801 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.897782    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4845995_0d75_4296_ad03_aea66ec32c37.slice/crio-a1143b100b31073216fb008a87ac72d0a957c5e5da17cc62da37cdc8ff2b1d92 WatchSource:0}: Error finding container a1143b100b31073216fb008a87ac72d0a957c5e5da17cc62da37cdc8ff2b1d92: Status 404 returned error can't find the container with id a1143b100b31073216fb008a87ac72d0a957c5e5da17cc62da37cdc8ff2b1d92
Jul 07 13:16:35.918072 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.918048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4srzl"
Jul 07 13:16:35.925579 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.925562    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5d83cc_f411_41ea_b874_b2f5aeb7426c.slice/crio-061339393f8e6b43861e348eb6c9569470c94cd86719180c4ffd577b17060b39 WatchSource:0}: Error finding container 061339393f8e6b43861e348eb6c9569470c94cd86719180c4ffd577b17060b39: Status 404 returned error can't find the container with id 061339393f8e6b43861e348eb6c9569470c94cd86719180c4ffd577b17060b39
Jul 07 13:16:35.929379 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.929365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvsmw"
Jul 07 13:16:35.935813 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.935792    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b21091_8600_485e_86d6_2b0293bc48ae.slice/crio-7d27e3ecab3340873c664f8255de81926a95bd51256dc8bef2bb2141b90dcebd WatchSource:0}: Error finding container 7d27e3ecab3340873c664f8255de81926a95bd51256dc8bef2bb2141b90dcebd: Status 404 returned error can't find the container with id 7d27e3ecab3340873c664f8255de81926a95bd51256dc8bef2bb2141b90dcebd
Jul 07 13:16:35.947273 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.947256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:16:35.954216 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.954197    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60e128f_b64e_4026_b937_2f7d61b35524.slice/crio-e2a51626210c312c747d387f38768c1f66c06b84d126beca059012e05d4b04ee WatchSource:0}: Error finding container e2a51626210c312c747d387f38768c1f66c06b84d126beca059012e05d4b04ee: Status 404 returned error can't find the container with id e2a51626210c312c747d387f38768c1f66c06b84d126beca059012e05d4b04ee
Jul 07 13:16:35.963517 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.963504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:16:35.970344 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.970325    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a209626_ef8e_403b_b8e4_4778a11c9e2d.slice/crio-2c8a381c95a31ddd43f9980d65f7afa8a1bfe42b0e5c967021b92b8f37a5ef14 WatchSource:0}: Error finding container 2c8a381c95a31ddd43f9980d65f7afa8a1bfe42b0e5c967021b92b8f37a5ef14: Status 404 returned error can't find the container with id 2c8a381c95a31ddd43f9980d65f7afa8a1bfe42b0e5c967021b92b8f37a5ef14
Jul 07 13:16:35.985099 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.985078    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7"
Jul 07 13:16:35.991433 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:35.991416    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1483d8_866d_4145_a97a_ec743ae7395d.slice/crio-f0f3d4889709ec1ac1765125c9ff316f04087235fd97e68659454b00a50e4636 WatchSource:0}: Error finding container f0f3d4889709ec1ac1765125c9ff316f04087235fd97e68659454b00a50e4636: Status 404 returned error can't find the container with id f0f3d4889709ec1ac1765125c9ff316f04087235fd97e68659454b00a50e4636
Jul 07 13:16:35.991921 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.991908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ksnhj"
Jul 07 13:16:35.999549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:35.999529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9bz7"
Jul 07 13:16:36.000107 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:36.000080    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20eff8d_cbbe_4869_b380_13a25e5fabd0.slice/crio-b0a81e79f37da4ac66f6d3eb76db38d2a234f82f4b33ec8d585da8fcc071c2af WatchSource:0}: Error finding container b0a81e79f37da4ac66f6d3eb76db38d2a234f82f4b33ec8d585da8fcc071c2af: Status 404 returned error can't find the container with id b0a81e79f37da4ac66f6d3eb76db38d2a234f82f4b33ec8d585da8fcc071c2af
Jul 07 13:16:36.006711 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:16:36.006688    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9315e0_f682_4632_acf8_0574443059e8.slice/crio-37b4e2d77fddad6831971f3f4f556e3f0e201319f9e2b09eb8a30a51f1a2fae0 WatchSource:0}: Error finding container 37b4e2d77fddad6831971f3f4f556e3f0e201319f9e2b09eb8a30a51f1a2fae0: Status 404 returned error can't find the container with id 37b4e2d77fddad6831971f3f4f556e3f0e201319f9e2b09eb8a30a51f1a2fae0
Jul 07 13:16:36.044471 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.044452    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 13:16:36.276740 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.275972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:36.276740 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.276026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:36.276740 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.276179    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:36.276740 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.276235    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:37.276217346 +0000 UTC m=+3.156588630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:36.276740 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.276617    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:36.276740 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.276664    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:37.276650107 +0000 UTC m=+3.157021392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:36.377438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.377403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:36.377646 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.377627    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:16:36.377708 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.377655    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:16:36.377708 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.377668    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:36.377808 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.377729    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:16:37.37770927 +0000 UTC m=+3.258080585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:36.603494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.603294    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 13:11:35 +0000 UTC" deadline="2028-02-29 17:51:25.736357883 +0000 UTC"
Jul 07 13:16:36.603494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.603331    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14452h34m49.133031465s"
Jul 07 13:16:36.757326 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.757294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:36.757496 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:36.757441    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:36.785702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.785641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerStarted","Data":"37b4e2d77fddad6831971f3f4f556e3f0e201319f9e2b09eb8a30a51f1a2fae0"}
Jul 07 13:16:36.798082 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.798044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ksnhj" event={"ID":"b20eff8d-cbbe-4869-b380-13a25e5fabd0","Type":"ContainerStarted","Data":"b0a81e79f37da4ac66f6d3eb76db38d2a234f82f4b33ec8d585da8fcc071c2af"}
Jul 07 13:16:36.807487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.807459    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 13:16:36.824277 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.824177    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdq6p" event={"ID":"2a209626-ef8e-403b-b8e4-4778a11c9e2d","Type":"ContainerStarted","Data":"2c8a381c95a31ddd43f9980d65f7afa8a1bfe42b0e5c967021b92b8f37a5ef14"}
Jul 07 13:16:36.841171 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.841123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"e2a51626210c312c747d387f38768c1f66c06b84d126beca059012e05d4b04ee"}
Jul 07 13:16:36.855348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.855265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4srzl" event={"ID":"4a5d83cc-f411-41ea-b874-b2f5aeb7426c","Type":"ContainerStarted","Data":"061339393f8e6b43861e348eb6c9569470c94cd86719180c4ffd577b17060b39"}
Jul 07 13:16:36.873658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.873626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c48gn" event={"ID":"a4845995-0d75-4296-ad03-aea66ec32c37","Type":"ContainerStarted","Data":"a1143b100b31073216fb008a87ac72d0a957c5e5da17cc62da37cdc8ff2b1d92"}
Jul 07 13:16:36.897412 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.897375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qdpl2" event={"ID":"635424cd-2fa4-482b-bfce-2952febc14e3","Type":"ContainerStarted","Data":"9ba59b6cc80cb366e6d11456e19cbc03cf9881a8d863e61376a738212b1d9c7b"}
Jul 07 13:16:36.915422 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.915390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7" event={"ID":"ac1483d8-866d-4145-a97a-ec743ae7395d","Type":"ContainerStarted","Data":"f0f3d4889709ec1ac1765125c9ff316f04087235fd97e68659454b00a50e4636"}
Jul 07 13:16:36.930075 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.930041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvsmw" event={"ID":"66b21091-8600-485e-86d6-2b0293bc48ae","Type":"ContainerStarted","Data":"7d27e3ecab3340873c664f8255de81926a95bd51256dc8bef2bb2141b90dcebd"}
Jul 07 13:16:36.945015 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.944964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal" event={"ID":"f2a914bd14bade07bf5d2dc3674ec896","Type":"ContainerStarted","Data":"5bd861d6f84af0e3450388ac002d9ce6dac0eeb5d5aae16fd2fac8a04ba52dc6"}
Jul 07 13:16:36.976023 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:36.975992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal" event={"ID":"a8cad6aa2145f5ea3bbbbae2bfaaf983","Type":"ContainerStarted","Data":"fb8622be14c2ef5452110ab247ded019d4a5e4c9133b1118ddae824d4d84e1cc"}
Jul 07 13:16:37.218265 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.218184    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 13:16:37.283078 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.283004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:37.283283 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.283084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:37.283283 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.283219    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:37.283283 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.283277    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:39.283260062 +0000 UTC m=+5.163631353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:37.283672 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.283652    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:37.283730 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.283708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:39.283692921 +0000 UTC m=+5.164064200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:37.384041 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.384018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:37.384154 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.384125    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:16:37.384154 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.384139    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:16:37.384154 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.384148    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:37.384343 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.384220    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:16:39.384202302 +0000 UTC m=+5.264573578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:37.604368 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.604265    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 13:11:35 +0000 UTC" deadline="2028-02-05 10:16:20.950287286 +0000 UTC"
Jul 07 13:16:37.604368 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.604323    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13868h59m43.34596854s"
Jul 07 13:16:37.754418 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.754154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:37.754588 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:37.754439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:37.754588 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.754533    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:37.754984 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:37.754957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:38.756266 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:38.756240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:38.756720 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:38.756374    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:39.303723 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:39.303627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:39.303723 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:39.303685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:39.303963 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.303826    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:39.303963 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.303886    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:43.303867805 +0000 UTC m=+9.184239080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:39.304499 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.304265    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:39.304499 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.304310    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:43.3042992 +0000 UTC m=+9.184670472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:39.404372 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:39.404332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:39.404533 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.404496    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:16:39.404533 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.404515    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:16:39.404533 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.404526    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:39.404673 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.404580    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:16:43.404563456 +0000 UTC m=+9.284934742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:39.754949 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:39.754875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:39.755094 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.755005    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:39.755094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:39.755044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:39.755222 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:39.755175    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:40.756646 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:40.756139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:40.756646 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:40.756296    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:41.755099 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:41.754588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:41.755099 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:41.754739    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:41.755099 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:41.754789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:41.755099 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:41.754912    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:42.754581 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:42.754486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:42.755140 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:42.754624    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:43.339519 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:43.339470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:43.339684 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:43.339562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:43.339684 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.339643    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:43.339684 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.339653    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:43.339816 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.339708    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:51.339691176 +0000 UTC m=+17.220062449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:43.339816 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.339734    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:16:51.339715455 +0000 UTC m=+17.220086742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:43.440046 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:43.440011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:43.440233 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.440210    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:16:43.440307 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.440238    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:16:43.440307 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.440251    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:43.440398 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.440309    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:16:51.440290814 +0000 UTC m=+17.320662101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:43.754961 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:43.754889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:43.755388 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.755001    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:43.755388 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:43.755066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:43.755388 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:43.755200    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:44.755028 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:44.754996    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:44.755409 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:44.755121    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:45.755026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:45.754989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:45.755215 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:45.755036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:45.755215 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:45.755109    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:45.755615 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:45.755265    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:46.757686 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:46.757659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:46.758047 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:46.757772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:47.754263 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:47.754224    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:47.754402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:47.754234    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:47.754402 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:47.754324    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:47.754522 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:47.754396    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:48.754473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:48.754441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:48.754909 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:48.754563    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:49.754985 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:49.754954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:49.755393 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:49.754954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:49.755393 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:49.755082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:49.755393 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:49.755153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:50.754949 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:50.754917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:50.755119 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:50.755059    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:51.399987 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:51.399953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:51.400235 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:51.400005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:51.400235 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.400116    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:51.400235 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.400186    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:51.400235 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.400199    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:07.400181763 +0000 UTC m=+33.280553048 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:16:51.400402 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.400250    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:07.400231686 +0000 UTC m=+33.280602961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:16:51.500376 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:51.500346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:51.500554 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.500533    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:16:51.500615 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.500561    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:16:51.500615 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.500572    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:51.500694 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.500625    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:17:07.500605791 +0000 UTC m=+33.380977081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:16:51.754511 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:51.754477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:51.754511 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:51.754496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:51.754733 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.754594    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:51.754733 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:51.754705    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:52.754779 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:52.754741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:52.755320 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:52.754878    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:53.754441 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:53.754401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:53.754441 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:53.754439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:53.754674 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:53.754544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:53.754674 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:53.754625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:54.755577 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:54.755547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:54.756033 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:54.755662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:55.754391 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:55.754357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:55.754575 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:55.754357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:55.754575 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:55.754465    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:55.754575 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:55.754545    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:56.754513 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:56.754347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:56.755311 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:56.754608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:57.017701 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.017657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qdpl2" event={"ID":"635424cd-2fa4-482b-bfce-2952febc14e3","Type":"ContainerStarted","Data":"0e76bbcaca8c1afa825c37f8419889182aa249e01608679a0f51b4bd7928d861"}
Jul 07 13:16:57.019119 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.019084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal" event={"ID":"a8cad6aa2145f5ea3bbbbae2bfaaf983","Type":"ContainerStarted","Data":"84c4a0a0f1e0f326c485e6f42bbc6a8df61fc56a1dda94092b4967c59821bda5"}
Jul 07 13:16:57.022919 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.022888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"92cc4705f7411641ea15339451f5a5328d33731a36080d5f1c8bed1b35a13016"}
Jul 07 13:16:57.023068 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.023048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"bd0f8605943f4129458a62fb774608471ae45d8834b08da65ba96f0c55e691cf"}
Jul 07 13:16:57.023144 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.023130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"ebf709b273c85257cff2e5ff172d0cd87516644d05ab24e7d3a442b7a2a7dfa1"}
Jul 07 13:16:57.023250 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.023235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"4947fd2d103db2f5f9f36b0015a5e53a3c7adae5f863be5120e02b7001438ff8"}
Jul 07 13:16:57.023316 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.023304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"ceb7b82bd4ae3dcf2e06fd5d64c4ae14edce7c89e792d6028a8d1ce46a816300"}
Jul 07 13:16:57.023382 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.023370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"d434a1e3e97dd275286c0f121c0814c5f59e23416058c1882674584747383ecb"}
Jul 07 13:16:57.029489 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.029471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4srzl" event={"ID":"4a5d83cc-f411-41ea-b874-b2f5aeb7426c","Type":"ContainerStarted","Data":"0b543c56d84783b2119075a47cc35a81bdebefc58d65223d5a27cee087526b25"}
Jul 07 13:16:57.035769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.035722    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qdpl2" podStartSLOduration=2.986136157 podStartE2EDuration="23.035712282s" totalImagesPullingTime="20.049576125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:57.035370182 +0000 UTC m=+22.915741477" watchObservedRunningTime="2026-07-07 13:16:57.035712282 +0000 UTC m=+22.916083576"
Jul 07 13:16:57.052603 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.052571    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4srzl" podStartSLOduration=3.010150101 podStartE2EDuration="23.052560102s" totalImagesPullingTime="20.042410001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:57.052321932 +0000 UTC m=+22.932693228" watchObservedRunningTime="2026-07-07 13:16:57.052560102 +0000 UTC m=+22.932931396"
Jul 07 13:16:57.066866 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.066776    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-157.ec2.internal" podStartSLOduration=22.066760544 podStartE2EDuration="22.066760544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:57.066386065 +0000 UTC m=+22.946757379" watchObservedRunningTime="2026-07-07 13:16:57.066760544 +0000 UTC m=+22.947131840"
Jul 07 13:16:57.754508 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.754328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:57.754853 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:57.754354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:57.754853 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:57.754584    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:16:57.754853 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:57.754660    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:58.033011 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.032938    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e9315e0-f682-4632-acf8-0574443059e8" containerID="3516405c9b4209a1e6c43ea3f09dffb241f2d7d5d5901c6b8b3045384b0b4925" exitCode=0
Jul 07 13:16:58.033275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.033011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerDied","Data":"3516405c9b4209a1e6c43ea3f09dffb241f2d7d5d5901c6b8b3045384b0b4925"}
Jul 07 13:16:58.034564 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.034495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ksnhj" event={"ID":"b20eff8d-cbbe-4869-b380-13a25e5fabd0","Type":"ContainerStarted","Data":"364cec6668d706415be0975f738b593d4d30761d3cd409bed251661cda5b3fb2"}
Jul 07 13:16:58.035976 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.035947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdq6p" event={"ID":"2a209626-ef8e-403b-b8e4-4778a11c9e2d","Type":"ContainerStarted","Data":"3b93158a09f060cb48ada2376500d6764c54d25ff794753996b0c0ea95eaf30c"}
Jul 07 13:16:58.037213 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.037191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c48gn" event={"ID":"a4845995-0d75-4296-ad03-aea66ec32c37","Type":"ContainerStarted","Data":"17819bca4e8afad0d183a07976f3104c0141033fbf1a77b4043b720bb9de4c73"}
Jul 07 13:16:58.038547 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.038522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7" event={"ID":"ac1483d8-866d-4145-a97a-ec743ae7395d","Type":"ContainerStarted","Data":"79d6673bc8e05b0c430f41a4d6a63358af77c4acff0fbc82907dd746a36eb693"}
Jul 07 13:16:58.040207 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.040186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvsmw" event={"ID":"66b21091-8600-485e-86d6-2b0293bc48ae","Type":"ContainerStarted","Data":"7e1e3a09d93f7c4dc298ad93888eac05131a855750fbe17a7c821c5ce5e18641"}
Jul 07 13:16:58.041614 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.041593    2581 generic.go:356] "Generic (PLEG): container finished" podID="f2a914bd14bade07bf5d2dc3674ec896" containerID="4fcd057f57c9e081e707fa21aa147b0da502860e9745b1a7d188eb34777eab7f" exitCode=0
Jul 07 13:16:58.041762 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.041739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal" event={"ID":"f2a914bd14bade07bf5d2dc3674ec896","Type":"ContainerDied","Data":"4fcd057f57c9e081e707fa21aa147b0da502860e9745b1a7d188eb34777eab7f"}
Jul 07 13:16:58.097071 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.097010    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bdq6p" podStartSLOduration=4.133125418 podStartE2EDuration="24.096996231s" totalImagesPullingTime="19.963870813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:58.08322675 +0000 UTC m=+23.963598071" watchObservedRunningTime="2026-07-07 13:16:58.096996231 +0000 UTC m=+23.977367525"
Jul 07 13:16:58.097280 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.097186    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ksnhj" podStartSLOduration=4.162921735 podStartE2EDuration="24.09717713s" totalImagesPullingTime="19.934255395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:58.097082068 +0000 UTC m=+23.977453379" watchObservedRunningTime="2026-07-07 13:16:58.09717713 +0000 UTC m=+23.977548417"
Jul 07 13:16:58.112191 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.112121    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c48gn" podStartSLOduration=4.075056714 podStartE2EDuration="24.112105168s" totalImagesPullingTime="20.037048454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:58.111902996 +0000 UTC m=+23.992274290" watchObservedRunningTime="2026-07-07 13:16:58.112105168 +0000 UTC m=+23.992476461"
Jul 07 13:16:58.125915 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.125864    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tvsmw" podStartSLOduration=4.127574582 podStartE2EDuration="24.12584698s" totalImagesPullingTime="19.998272398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:58.125535556 +0000 UTC m=+24.005906849" watchObservedRunningTime="2026-07-07 13:16:58.12584698 +0000 UTC m=+24.006218276"
Jul 07 13:16:58.192687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.192651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:16:58.260267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.260245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:16:58.314707 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.314686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:16:58.525802 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.525781    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 13:16:58.633969 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.633865    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T13:16:58.525796604Z","UUID":"06d9df85-02dd-4fcb-a0fd-f13813dc41c3","Handler":null,"Name":"","Endpoint":""}
Jul 07 13:16:58.635199 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.635183    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 13:16:58.635283 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.635209    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 13:16:58.754928 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:58.754902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:16:58.755453 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:58.754999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:16:59.045672 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:59.045641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal" event={"ID":"f2a914bd14bade07bf5d2dc3674ec896","Type":"ContainerStarted","Data":"2292ee3fb9471ac5cc85c0c3f85227c1cca03c9a132f99ed8b848152eea13cd5"}
Jul 07 13:16:59.048254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:59.048228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7" event={"ID":"ac1483d8-866d-4145-a97a-ec743ae7395d","Type":"ContainerStarted","Data":"30a26f62f6040439100beb4e2778677aced735b7086bc25bdc82cb79e58978b2"}
Jul 07 13:16:59.059124 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:59.059085    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-157.ec2.internal" podStartSLOduration=24.059071535 podStartE2EDuration="24.059071535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:16:59.058770456 +0000 UTC m=+24.939141749" watchObservedRunningTime="2026-07-07 13:16:59.059071535 +0000 UTC m=+24.939442829"
Jul 07 13:16:59.754937 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:59.754914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:16:59.755271 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:16:59.754914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:16:59.755271 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:59.755013    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:16:59.755271 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:16:59.755092    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:17:00.053785 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:00.053748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"d794e7702c590fafee0c8030c5b43257e25e8d8b19cfb0cc87cf97a2f0182e64"}
Jul 07 13:17:00.055759 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:00.055722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7" event={"ID":"ac1483d8-866d-4145-a97a-ec743ae7395d","Type":"ContainerStarted","Data":"06b555875c5498d4d4de568ecba1959ae8b3bae37ca59ee896b2c57739717691"}
Jul 07 13:17:00.073648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:00.073584    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sg4f7" podStartSLOduration=3.847559197 podStartE2EDuration="26.073566329s" totalImagesPullingTime="22.226007132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:00.072986633 +0000 UTC m=+25.953357928" watchObservedRunningTime="2026-07-07 13:17:00.073566329 +0000 UTC m=+25.953937623"
Jul 07 13:17:00.419484 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:00.419395    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:17:00.420137 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:00.420112    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:17:00.755293 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:00.755101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:00.755740 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:00.755392    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:17:01.058618 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.058528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:17:01.059676 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.059644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdq6p"
Jul 07 13:17:01.120132 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.120108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:01.753999 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.753965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:01.754248 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.753965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:01.754248 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:01.754068    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:17:01.754248 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:01.754191    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:17:01.773745 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.773719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:01.871054 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:01.871011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:02.064577 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:02.064499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" event={"ID":"b60e128f-b64e-4026-b937-2f7d61b35524","Type":"ContainerStarted","Data":"fb399deb6d60c90de2f1bd1b9a407dd675cfd29cb27be41a2638fe7e14b2b5e5"}
Jul 07 13:17:02.090232 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:02.090190    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k" podStartSLOduration=7.841968785 podStartE2EDuration="28.090175448s" totalImagesPullingTime="20.248206663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:02.089900766 +0000 UTC m=+27.970272085" watchObservedRunningTime="2026-07-07 13:17:02.090175448 +0000 UTC m=+27.970546732"
Jul 07 13:17:02.754753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:02.754709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:02.754923 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:02.754844    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:17:03.066748 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.066481    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:17:03.067129 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.066869    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:17:03.067129 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.066939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:17:03.084614 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.084595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:17:03.085833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.085817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:17:03.198807 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.198745    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s6p2q"]
Jul 07 13:17:03.198961 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.198829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:03.198961 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:03.198908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:17:03.206175 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.206141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lxbcn"]
Jul 07 13:17:03.206295 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.206249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:03.206347 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:03.206326    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:17:03.218102 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.218081    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbpsm"]
Jul 07 13:17:03.218195 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.218155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:03.218265 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:03.218239    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:17:03.378654 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.378623    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:03.479366 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:03.479343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:04.069944 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:04.069908    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e9315e0-f682-4632-acf8-0574443059e8" containerID="1f4a0b39a31e278ad2d450f8b37b3fe4d625f4b86b3508a5574ad8475ba808b8" exitCode=0
Jul 07 13:17:04.070412 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:04.069950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerDied","Data":"1f4a0b39a31e278ad2d450f8b37b3fe4d625f4b86b3508a5574ad8475ba808b8"}
Jul 07 13:17:04.755447 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:04.755419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:04.755578 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:04.755515    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:17:04.755578 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:04.755534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:04.755654 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:04.755631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:04.755741 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:04.755722    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:17:04.755811 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:04.755794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:17:06.076588 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.076556    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e9315e0-f682-4632-acf8-0574443059e8" containerID="3c916f882513f751851a32d82d1a8245e2b9dee10b54b0b74be51eec5ab27888" exitCode=0
Jul 07 13:17:06.076922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.076607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerDied","Data":"3c916f882513f751851a32d82d1a8245e2b9dee10b54b0b74be51eec5ab27888"}
Jul 07 13:17:06.736955 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.736919    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:06.754907 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.754881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:06.755062 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.754892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:06.755062 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:06.754989    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:17:06.755062 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.755004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:06.755544 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:06.755518    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:17:06.756930 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:06.755634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:17:06.835936 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:06.835902    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:07.423077 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.423013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:07.423430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.423058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:07.423430 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.423197    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:17:07.423430 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.423203    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 13:17:07.423430 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.423251    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs podName:404f302b-7cf7-41ff-a9d1-a2ada86d1ab5 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:39.423236436 +0000 UTC m=+65.303607712 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs") pod "network-metrics-daemon-s6p2q" (UID: "404f302b-7cf7-41ff-a9d1-a2ada86d1ab5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 13:17:07.423430 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.423264    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret podName:81f506f0-9be6-48c3-b771-9e7648ada7a4 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:39.42325839 +0000 UTC m=+65.303629665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret") pod "global-pull-secret-syncer-mbpsm" (UID: "81f506f0-9be6-48c3-b771-9e7648ada7a4") : object "kube-system"/"original-pull-secret" not registered
Jul 07 13:17:07.523732 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.523706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:07.523862 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.523847    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 13:17:07.523896 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.523868    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 13:17:07.523896 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.523877    2581 projected.go:196] Error preparing data for projected volume kube-api-access-wfgnp for pod openshift-network-diagnostics/network-check-target-lxbcn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:17:07.523975 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:07.523918    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp podName:8f23e9b4-4c44-4420-aa9c-26e57abc4f7c nodeName:}" failed. No retries permitted until 2026-07-07 13:17:39.523905969 +0000 UTC m=+65.404277241 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfgnp" (UniqueName: "kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp") pod "network-check-target-lxbcn" (UID: "8f23e9b4-4c44-4420-aa9c-26e57abc4f7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 13:17:07.537824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.537794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:07.620824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.620785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:07.690182 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.690135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:07.780235 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.780198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:07.880594 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:07.880567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:08.083411 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.083385    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e9315e0-f682-4632-acf8-0574443059e8" containerID="ee5382f879ccae6201ecf2bbb431bd8ac5eac201ccf39fa74f5b662f6b9f6601" exitCode=0
Jul 07 13:17:08.083553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.083434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerDied","Data":"ee5382f879ccae6201ecf2bbb431bd8ac5eac201ccf39fa74f5b662f6b9f6601"}
Jul 07 13:17:08.209454 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.209429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:08.310080 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.310054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:08.511421 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.511393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:08.583060 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.583021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:08.641141 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.641089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:08.754983 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.754961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:08.755133 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.754998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:08.755133 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.754961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:08.755133 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:08.755087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s6p2q" podUID="404f302b-7cf7-41ff-a9d1-a2ada86d1ab5"
Jul 07 13:17:08.755323 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:08.755180    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lxbcn" podUID="8f23e9b4-4c44-4420-aa9c-26e57abc4f7c"
Jul 07 13:17:08.755323 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:08.755265    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mbpsm" podUID="81f506f0-9be6-48c3-b771-9e7648ada7a4"
Jul 07 13:17:08.922524 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:08.922449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:09.015545 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.015520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:09.103468 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.103435    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:09.209901 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.209821    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c48gn_a4845995-0d75-4296-ad03-aea66ec32c37/dns-node-resolver/0.log"
Jul 07 13:17:09.720599 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.720567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:09.785522 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.785495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:09.849357 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.849334    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-157.ec2.internal" event="NodeReady"
Jul 07 13:17:09.849481 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.849452    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 13:17:09.891711 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.891652    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hbz7c"]
Jul 07 13:17:09.906032 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.906009    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tnwlb"]
Jul 07 13:17:09.906200 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.906137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:09.910711 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.910681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nmbgg\""
Jul 07 13:17:09.910711 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.910707    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 13:17:09.910896 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.910874    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 13:17:09.915774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.914720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnwlb"]
Jul 07 13:17:09.915774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.914759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hbz7c"]
Jul 07 13:17:09.915774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.914787    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2fvnk"]
Jul 07 13:17:09.915774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.914861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:09.917863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.917840    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9d22t\""
Jul 07 13:17:09.918069 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.918054    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 13:17:09.918188 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.918088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 13:17:09.918276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.918237    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 13:17:09.918455 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.918441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:09.920836 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.920807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 13:17:09.920836 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.920823    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 13:17:09.920983 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.920847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nfz7b\""
Jul 07 13:17:09.920983 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.920847    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 13:17:09.921195 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.921181    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 13:17:09.929348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:09.929323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2fvnk"]
Jul 07 13:17:10.009936 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.009919    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ksnhj_b20eff8d-cbbe-4869-b380-13a25e5fabd0/node-ca/0.log"
Jul 07 13:17:10.014493 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.014473    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:10.041636 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9e457f-cf36-4628-8e9f-9496d8111b71-config-volume\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.041742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd2f7\" (UniqueName: \"kubernetes.io/projected/7b9e457f-cf36-4628-8e9f-9496d8111b71-kube-api-access-dd2f7\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.041742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9e457f-cf36-4628-8e9f-9496d8111b71-tmp-dir\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.041742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2-cert\") pod \"ingress-canary-tnwlb\" (UID: \"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2\") " pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.041742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.041952 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.041952 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.041952 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-data-volume\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.041952 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvg77\" (UniqueName: \"kubernetes.io/projected/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-kube-api-access-qvg77\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.041952 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-crio-socket\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.041952 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.041923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6c68\" (UniqueName: \"kubernetes.io/projected/bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2-kube-api-access-p6c68\") pod \"ingress-canary-tnwlb\" (UID: \"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2\") " pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.116352 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.116317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:10.142640 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9e457f-cf36-4628-8e9f-9496d8111b71-tmp-dir\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.142640 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2-cert\") pod \"ingress-canary-tnwlb\" (UID: \"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2\") " pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.142819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.142819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.142819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.142819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-data-volume\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.142860    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.142874    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.142931    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls podName:7b9e457f-cf36-4628-8e9f-9496d8111b71 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:10.642914471 +0000 UTC m=+36.523285743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls") pod "dns-default-hbz7c" (UID: "7b9e457f-cf36-4628-8e9f-9496d8111b71") : secret "dns-default-metrics-tls" not found
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvg77\" (UniqueName: \"kubernetes.io/projected/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-kube-api-access-qvg77\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.142950    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls podName:4c9f9d2c-f1b8-41aa-af80-7854ce25f309 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:10.642939689 +0000 UTC m=+36.523310961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fvnk" (UID: "4c9f9d2c-f1b8-41aa-af80-7854ce25f309") : secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142952    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9e457f-cf36-4628-8e9f-9496d8111b71-tmp-dir\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.142985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-crio-socket\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.143010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6c68\" (UniqueName: \"kubernetes.io/projected/bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2-kube-api-access-p6c68\") pod \"ingress-canary-tnwlb\" (UID: \"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2\") " pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.143407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143051    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-data-volume\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.143407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9e457f-cf36-4628-8e9f-9496d8111b71-config-volume\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.143407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd2f7\" (UniqueName: \"kubernetes.io/projected/7b9e457f-cf36-4628-8e9f-9496d8111b71-kube-api-access-dd2f7\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.143407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143210    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-crio-socket\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.143407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143237    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.143555 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.143541    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b9e457f-cf36-4628-8e9f-9496d8111b71-config-volume\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.147002 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.146981    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2-cert\") pod \"ingress-canary-tnwlb\" (UID: \"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2\") " pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.151620 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.151601    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvg77\" (UniqueName: \"kubernetes.io/projected/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-kube-api-access-qvg77\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.152547 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.152527    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6c68\" (UniqueName: \"kubernetes.io/projected/bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2-kube-api-access-p6c68\") pod \"ingress-canary-tnwlb\" (UID: \"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2\") " pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.152718 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.152703    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd2f7\" (UniqueName: \"kubernetes.io/projected/7b9e457f-cf36-4628-8e9f-9496d8111b71-kube-api-access-dd2f7\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.229919 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.229901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnwlb"
Jul 07 13:17:10.427753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.427686    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnwlb"]
Jul 07 13:17:10.432582 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:10.432554    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf83232b_8e51_4f9d_8c9e_b9d5b68e94e2.slice/crio-66dbce213b50d0e1ca318b287a7c29dcc9d97d00990a341eec3827e708a61a69 WatchSource:0}: Error finding container 66dbce213b50d0e1ca318b287a7c29dcc9d97d00990a341eec3827e708a61a69: Status 404 returned error can't find the container with id 66dbce213b50d0e1ca318b287a7c29dcc9d97d00990a341eec3827e708a61a69
Jul 07 13:17:10.508354 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.508323    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5fxfw"]
Jul 07 13:17:10.518101 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.518077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.520028 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.519832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5fxfw"]
Jul 07 13:17:10.520881 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.520862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wftv7\""
Jul 07 13:17:10.520979 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.520879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 13:17:10.521139 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.521125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 13:17:10.521916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.521898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 13:17:10.522008 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.521917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 13:17:10.551699 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.551673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:10.620518 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.620491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:10.648771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.648749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4edfe42-8503-4c64-be16-f2544ffd8bb2-signing-key\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.648883 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.648774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4edfe42-8503-4c64-be16-f2544ffd8bb2-signing-cabundle\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.648883 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.648808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwssq\" (UniqueName: \"kubernetes.io/projected/a4edfe42-8503-4c64-be16-f2544ffd8bb2-kube-api-access-gwssq\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.648957 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.648937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:10.648990 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.648973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:10.649087 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.649072    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 13:17:10.649127 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.649105    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:10.649184 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.649125    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls podName:7b9e457f-cf36-4628-8e9f-9496d8111b71 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:11.64911008 +0000 UTC m=+37.529481352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls") pod "dns-default-hbz7c" (UID: "7b9e457f-cf36-4628-8e9f-9496d8111b71") : secret "dns-default-metrics-tls" not found
Jul 07 13:17:10.649238 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:10.649185    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls podName:4c9f9d2c-f1b8-41aa-af80-7854ce25f309 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:11.649147606 +0000 UTC m=+37.529518883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fvnk" (UID: "4c9f9d2c-f1b8-41aa-af80-7854ce25f309") : secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:10.749679 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.749660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4edfe42-8503-4c64-be16-f2544ffd8bb2-signing-key\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.750447 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.749687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4edfe42-8503-4c64-be16-f2544ffd8bb2-signing-cabundle\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.750447 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.749755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwssq\" (UniqueName: \"kubernetes.io/projected/a4edfe42-8503-4c64-be16-f2544ffd8bb2-kube-api-access-gwssq\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.750589 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.750569    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4edfe42-8503-4c64-be16-f2544ffd8bb2-signing-cabundle\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.752912 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.752890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4edfe42-8503-4c64-be16-f2544ffd8bb2-signing-key\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.754951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.754932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:10.755080 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.754987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:10.755080 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.754992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:10.757467 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.757449    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 13:17:10.757885 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.757869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sgxmd\""
Jul 07 13:17:10.757959 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.757909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 13:17:10.758015 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.757871    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 13:17:10.758141 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.758125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 13:17:10.758223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.758140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9nksx\""
Jul 07 13:17:10.758343 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.758326    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwssq\" (UniqueName: \"kubernetes.io/projected/a4edfe42-8503-4c64-be16-f2544ffd8bb2-kube-api-access-gwssq\") pod \"service-ca-758f599c5-5fxfw\" (UID: \"a4edfe42-8503-4c64-be16-f2544ffd8bb2\") " pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.827872 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.827848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5fxfw"
Jul 07 13:17:10.968294 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:10.968268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5fxfw"]
Jul 07 13:17:10.971686 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:10.971650    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4edfe42_8503_4c64_be16_f2544ffd8bb2.slice/crio-99d3b8d0c17f4ef4068018884c793ff5786e19814a79d633bf382efeb50b941c WatchSource:0}: Error finding container 99d3b8d0c17f4ef4068018884c793ff5786e19814a79d633bf382efeb50b941c: Status 404 returned error can't find the container with id 99d3b8d0c17f4ef4068018884c793ff5786e19814a79d633bf382efeb50b941c
Jul 07 13:17:11.091570 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:11.091472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5fxfw" event={"ID":"a4edfe42-8503-4c64-be16-f2544ffd8bb2","Type":"ContainerStarted","Data":"99d3b8d0c17f4ef4068018884c793ff5786e19814a79d633bf382efeb50b941c"}
Jul 07 13:17:11.092470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:11.092451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnwlb" event={"ID":"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2","Type":"ContainerStarted","Data":"66dbce213b50d0e1ca318b287a7c29dcc9d97d00990a341eec3827e708a61a69"}
Jul 07 13:17:11.657407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:11.657372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:11.657407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:11.657417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:11.657648 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:11.657559    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:11.657648 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:11.657617    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls podName:4c9f9d2c-f1b8-41aa-af80-7854ce25f309 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:13.657602174 +0000 UTC m=+39.537973446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fvnk" (UID: "4c9f9d2c-f1b8-41aa-af80-7854ce25f309") : secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:11.657919 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:11.657888    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 13:17:11.658040 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:11.657960    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls podName:7b9e457f-cf36-4628-8e9f-9496d8111b71 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:13.657942078 +0000 UTC m=+39.538313362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls") pod "dns-default-hbz7c" (UID: "7b9e457f-cf36-4628-8e9f-9496d8111b71") : secret "dns-default-metrics-tls" not found
Jul 07 13:17:12.144645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:12.144604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:13.676275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:13.676240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:13.676275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:13.676281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:13.676835 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:13.676411    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 13:17:13.676835 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:13.676456    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:13.676835 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:13.676481    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls podName:7b9e457f-cf36-4628-8e9f-9496d8111b71 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:17.676463009 +0000 UTC m=+43.556834294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls") pod "dns-default-hbz7c" (UID: "7b9e457f-cf36-4628-8e9f-9496d8111b71") : secret "dns-default-metrics-tls" not found
Jul 07 13:17:13.676835 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:13.676510    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls podName:4c9f9d2c-f1b8-41aa-af80-7854ce25f309 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:17.676499093 +0000 UTC m=+43.556870364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fvnk" (UID: "4c9f9d2c-f1b8-41aa-af80-7854ce25f309") : secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:13.964274 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:13.964186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:14.210926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:14.210890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:14.691832 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:14.691786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:14.767339 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:14.767293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:16.107495 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.107459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnwlb" event={"ID":"bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2","Type":"ContainerStarted","Data":"ca11154f47f7d15887ba564a8d39f04e3c8cf4b56a21e8b24c259fd58d42f891"}
Jul 07 13:17:16.110378 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.110354    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e9315e0-f682-4632-acf8-0574443059e8" containerID="56b7d8f0c6f38332c7a0bbdebf9d7e8b4c5eb6f2bb18be7c2bafd1d04a61dff5" exitCode=0
Jul 07 13:17:16.110502 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.110441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerDied","Data":"56b7d8f0c6f38332c7a0bbdebf9d7e8b4c5eb6f2bb18be7c2bafd1d04a61dff5"}
Jul 07 13:17:16.112000 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.111976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5fxfw" event={"ID":"a4edfe42-8503-4c64-be16-f2544ffd8bb2","Type":"ContainerStarted","Data":"7dfbaa6ebd21bc12fcd06a9fc3709b7fdbd3ff99f01e71dcf3f1a91d8da4beaf"}
Jul 07 13:17:16.123491 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.123449    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tnwlb" podStartSLOduration=2.208480156 podStartE2EDuration="7.123438684s" totalImagesPullingTime="4.914958528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:16.123096571 +0000 UTC m=+42.003467865" watchObservedRunningTime="2026-07-07 13:17:16.123438684 +0000 UTC m=+42.003809979"
Jul 07 13:17:16.138494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.138449    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-5fxfw" podStartSLOduration=1.758044184 podStartE2EDuration="6.13843675s" totalImagesPullingTime="4.380392566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:16.137656051 +0000 UTC m=+42.018027356" watchObservedRunningTime="2026-07-07 13:17:16.13843675 +0000 UTC m=+42.018808043"
Jul 07 13:17:16.815134 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:16.815101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:17.118955 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.118874    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e9315e0-f682-4632-acf8-0574443059e8" containerID="0720e72ae69744b3b40a11787119ae86c714ba78cba498e071a78a5196c1581b" exitCode=0
Jul 07 13:17:17.119401 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.119002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerDied","Data":"0720e72ae69744b3b40a11787119ae86c714ba78cba498e071a78a5196c1581b"}
Jul 07 13:17:17.205301 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.204915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:17.313033 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.312971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:17.513272 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.513237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:17.623215 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.623186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:17.708035 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.708006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:17.708184 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.708046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:17.708241 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:17.708181    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:17.708241 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:17.708182    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 13:17:17.708241 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:17.708236    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls podName:4c9f9d2c-f1b8-41aa-af80-7854ce25f309 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:25.708218562 +0000 UTC m=+51.588589836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2fvnk" (UID: "4c9f9d2c-f1b8-41aa-af80-7854ce25f309") : secret "insights-runtime-extractor-tls" not found
Jul 07 13:17:17.708408 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:17.708253    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls podName:7b9e457f-cf36-4628-8e9f-9496d8111b71 nodeName:}" failed. No retries permitted until 2026-07-07 13:17:25.708244009 +0000 UTC m=+51.588615286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls") pod "dns-default-hbz7c" (UID: "7b9e457f-cf36-4628-8e9f-9496d8111b71") : secret "dns-default-metrics-tls" not found
Jul 07 13:17:17.731331 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.731300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:17.997285 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:17.997252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:18.124924 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:18.124893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" event={"ID":"7e9315e0-f682-4632-acf8-0574443059e8","Type":"ContainerStarted","Data":"c838949cd90602990436cb4ac4564ee43712e86f619c21cec27d70d8a98c3f0e"}
Jul 07 13:17:18.151312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:18.151264    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z9bz7" podStartSLOduration=9.27667854 podStartE2EDuration="44.151250438s" totalImagesPullingTime="34.874571898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:18.150974324 +0000 UTC m=+44.031345619" watchObservedRunningTime="2026-07-07 13:17:18.151250438 +0000 UTC m=+44.031621732"
Jul 07 13:17:18.792625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:18.792596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:18.908851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:18.908826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:19.429104 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:19.429073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:19.595721 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:19.595695    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:19.969483 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:19.969454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:20.216036 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:20.216000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:20.505578 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:20.505550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:20.606939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:20.606905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:20.666430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:20.666407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:21.169237 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:21.169212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:21.268120 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:21.268089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:21.760627 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:21.760589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:21.965607 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:21.965573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:22.082412 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:22.082338    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:22.172209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:22.172181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:22.227907 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:22.227881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:22.362920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:22.362861    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:22.493143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:22.493113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:22.593749 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:22.593718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:23.174492 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:23.174453    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:23.563834 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:23.563801    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:23.760407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:23.760377    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:23.959574 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:23.959505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:24.165472 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:24.165444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:24.357865 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:24.357835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:25.358705 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.358668    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:25.557980 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.557948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:25.764154 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.764127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:25.764397 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.764173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:25.767246 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.767216    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b9e457f-cf36-4628-8e9f-9496d8111b71-metrics-tls\") pod \"dns-default-hbz7c\" (UID: \"7b9e457f-cf36-4628-8e9f-9496d8111b71\") " pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:25.767342 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.767218    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4c9f9d2c-f1b8-41aa-af80-7854ce25f309-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2fvnk\" (UID: \"4c9f9d2c-f1b8-41aa-af80-7854ce25f309\") " pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:25.822078 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.822052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:25.840808 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.840788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2fvnk"
Jul 07 13:17:25.960984 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.960949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:25.967185 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:25.967127    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hbz7c"]
Jul 07 13:17:25.977345 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:25.977316    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9e457f_cf36_4628_8e9f_9496d8111b71.slice/crio-742479e17835089954ffec86fd829d4a4ffdc63dfc9eaa7d55227e88060b0db7 WatchSource:0}: Error finding container 742479e17835089954ffec86fd829d4a4ffdc63dfc9eaa7d55227e88060b0db7: Status 404 returned error can't find the container with id 742479e17835089954ffec86fd829d4a4ffdc63dfc9eaa7d55227e88060b0db7
Jul 07 13:17:26.001641 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:26.001622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2fvnk"]
Jul 07 13:17:26.004974 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:26.004952    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9f9d2c_f1b8_41aa_af80_7854ce25f309.slice/crio-a2819a2370f44c30643267bd5886c7d635a734c79049866fd26bece679506798 WatchSource:0}: Error finding container a2819a2370f44c30643267bd5886c7d635a734c79049866fd26bece679506798: Status 404 returned error can't find the container with id a2819a2370f44c30643267bd5886c7d635a734c79049866fd26bece679506798
Jul 07 13:17:26.144090 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:26.144011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hbz7c" event={"ID":"7b9e457f-cf36-4628-8e9f-9496d8111b71","Type":"ContainerStarted","Data":"742479e17835089954ffec86fd829d4a4ffdc63dfc9eaa7d55227e88060b0db7"}
Jul 07 13:17:26.145129 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:26.145105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fvnk" event={"ID":"4c9f9d2c-f1b8-41aa-af80-7854ce25f309","Type":"ContainerStarted","Data":"a2819a2370f44c30643267bd5886c7d635a734c79049866fd26bece679506798"}
Jul 07 13:17:26.163274 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:26.163252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:26.772836 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:26.772794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:26.960932 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:26.960854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:27.388357 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:27.388322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:27.567268 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:27.567234    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:30.298786 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:30.298746    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:30.399943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:30.399915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:30.500694 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:30.500669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:30.784438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:30.784358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:30.964012 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:30.963981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:31.162398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.162301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hbz7c" event={"ID":"7b9e457f-cf36-4628-8e9f-9496d8111b71","Type":"ContainerStarted","Data":"7b97e92b5c60ed8825f2845466eab3fce799dd8f99068ffe8afba9023e4d4103"}
Jul 07 13:17:31.162398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.162396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:31.162605 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.162424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hbz7c" event={"ID":"7b9e457f-cf36-4628-8e9f-9496d8111b71","Type":"ContainerStarted","Data":"5d72a2c0265bf489937e5ed44add30ae4f12cad9880bbddee4bef02a27b6ad8e"}
Jul 07 13:17:31.163895 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.163872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fvnk" event={"ID":"4c9f9d2c-f1b8-41aa-af80-7854ce25f309","Type":"ContainerStarted","Data":"2a32bf4f58269dcf4a046acb3511122b9483c94158f6227b6665fae731c24100"}
Jul 07 13:17:31.163986 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.163921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fvnk" event={"ID":"4c9f9d2c-f1b8-41aa-af80-7854ce25f309","Type":"ContainerStarted","Data":"6ba1d1e07f0e6130d3f6c0a88e2e0f6369afdbfafc920cf2e2a3b69d31f6231c"}
Jul 07 13:17:31.180855 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.180832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:31.181094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.181059    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hbz7c" podStartSLOduration=17.939992109 podStartE2EDuration="22.181046969s" totalImagesPullingTime="4.24105486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:31.17971643 +0000 UTC m=+57.060087744" watchObservedRunningTime="2026-07-07 13:17:31.181046969 +0000 UTC m=+57.061418262"
Jul 07 13:17:31.231741 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.231719    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:31.774720 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.774684    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:31.997192 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:31.997132    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:32.138084 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.137997    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"]
Jul 07 13:17:32.140244 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.140219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.144469 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.144448    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 13:17:32.145394 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.145372    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 13:17:32.145617 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.145600    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 13:17:32.145825 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.145810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 13:17:32.146037 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.146023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mccxx\""
Jul 07 13:17:32.154119 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.154098    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"]
Jul 07 13:17:32.215736 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.215617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dn6\" (UniqueName: \"kubernetes.io/projected/37a22f62-0d29-4c19-ba7b-4712ee5af95e-kube-api-access-l4dn6\") pod \"managed-serviceaccount-addon-agent-d58d654b-2kdlt\" (UID: \"37a22f62-0d29-4c19-ba7b-4712ee5af95e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.215736 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.215671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a22f62-0d29-4c19-ba7b-4712ee5af95e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d58d654b-2kdlt\" (UID: \"37a22f62-0d29-4c19-ba7b-4712ee5af95e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.226542 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.226509    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:32.235346 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.235321    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-kztcl"]
Jul 07 13:17:32.237594 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.237566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:32.240371 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.240351    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bddc7bbc5-khlbj"]
Jul 07 13:17:32.241022 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.241004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 13:17:32.241465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.241230    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 13:17:32.241465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.241304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wh8tx\""
Jul 07 13:17:32.242249 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.242225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.246207 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.246190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 13:17:32.246315 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.246236    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 13:17:32.246497 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.246481    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 13:17:32.246871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.246835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x9hsn\""
Jul 07 13:17:32.250894 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.250877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 13:17:32.256683 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.256660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-kztcl"]
Jul 07 13:17:32.257374 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.257354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bddc7bbc5-khlbj"]
Jul 07 13:17:32.316201 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc338116-d8e0-4bfd-a457-d2b7254994d2-image-registry-private-configuration\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc338116-d8e0-4bfd-a457-d2b7254994d2-ca-trust-extracted\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc338116-d8e0-4bfd-a457-d2b7254994d2-trusted-ca\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc338116-d8e0-4bfd-a457-d2b7254994d2-installation-pull-secrets\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-registry-tls\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-bound-sa-token\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759cp\" (UniqueName: \"kubernetes.io/projected/8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1-kube-api-access-759cp\") pod \"downloads-685bfb5f64-kztcl\" (UID: \"8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1\") " pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:32.316634 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hf2p\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-kube-api-access-8hf2p\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.316634 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dn6\" (UniqueName: \"kubernetes.io/projected/37a22f62-0d29-4c19-ba7b-4712ee5af95e-kube-api-access-l4dn6\") pod \"managed-serviceaccount-addon-agent-d58d654b-2kdlt\" (UID: \"37a22f62-0d29-4c19-ba7b-4712ee5af95e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.316634 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a22f62-0d29-4c19-ba7b-4712ee5af95e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d58d654b-2kdlt\" (UID: \"37a22f62-0d29-4c19-ba7b-4712ee5af95e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.316634 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.316549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc338116-d8e0-4bfd-a457-d2b7254994d2-registry-certificates\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.319487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.319466    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37a22f62-0d29-4c19-ba7b-4712ee5af95e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d58d654b-2kdlt\" (UID: \"37a22f62-0d29-4c19-ba7b-4712ee5af95e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.324920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.324885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:32.326626 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.326605    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dn6\" (UniqueName: \"kubernetes.io/projected/37a22f62-0d29-4c19-ba7b-4712ee5af95e-kube-api-access-l4dn6\") pod \"managed-serviceaccount-addon-agent-d58d654b-2kdlt\" (UID: \"37a22f62-0d29-4c19-ba7b-4712ee5af95e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.416966 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.416890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc338116-d8e0-4bfd-a457-d2b7254994d2-ca-trust-extracted\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.416966 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.416931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc338116-d8e0-4bfd-a457-d2b7254994d2-trusted-ca\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.416966 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.416950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc338116-d8e0-4bfd-a457-d2b7254994d2-installation-pull-secrets\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.417248 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.416974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-registry-tls\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.417248 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.416999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-bound-sa-token\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.417248 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.417027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759cp\" (UniqueName: \"kubernetes.io/projected/8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1-kube-api-access-759cp\") pod \"downloads-685bfb5f64-kztcl\" (UID: \"8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1\") " pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:32.417555 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.417072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hf2p\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-kube-api-access-8hf2p\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.417555 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.417530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc338116-d8e0-4bfd-a457-d2b7254994d2-registry-certificates\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.417664 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.417572    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc338116-d8e0-4bfd-a457-d2b7254994d2-ca-trust-extracted\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.417664 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.417576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc338116-d8e0-4bfd-a457-d2b7254994d2-image-registry-private-configuration\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.418402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.418286    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc338116-d8e0-4bfd-a457-d2b7254994d2-trusted-ca\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.419427 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.419397    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc338116-d8e0-4bfd-a457-d2b7254994d2-registry-certificates\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.422272 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.420813    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc338116-d8e0-4bfd-a457-d2b7254994d2-installation-pull-secrets\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.422272 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.420977    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc338116-d8e0-4bfd-a457-d2b7254994d2-image-registry-private-configuration\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.422272 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.421153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-registry-tls\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.429106 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.429078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:32.429359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.429340    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hf2p\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-kube-api-access-8hf2p\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.429698 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.429678    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc338116-d8e0-4bfd-a457-d2b7254994d2-bound-sa-token\") pod \"image-registry-bddc7bbc5-khlbj\" (UID: \"bc338116-d8e0-4bfd-a457-d2b7254994d2\") " pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:32.430470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.430449    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-759cp\" (UniqueName: \"kubernetes.io/projected/8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1-kube-api-access-759cp\") pod \"downloads-685bfb5f64-kztcl\" (UID: \"8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1\") " pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:32.461948 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.461924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"
Jul 07 13:17:32.523870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.523838    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:32.548620 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.548588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:32.554660 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:32.554641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:33.468489 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:33.468456    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bddc7bbc5-khlbj"]
Jul 07 13:17:33.471731 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:33.471704    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc338116_d8e0_4bfd_a457_d2b7254994d2.slice/crio-fe4af6f71a195a5129adfab2a24ace795b781c7f64afac46ed30232e38200d84 WatchSource:0}: Error finding container fe4af6f71a195a5129adfab2a24ace795b781c7f64afac46ed30232e38200d84: Status 404 returned error can't find the container with id fe4af6f71a195a5129adfab2a24ace795b781c7f64afac46ed30232e38200d84
Jul 07 13:17:33.688423 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:33.688344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-kztcl"]
Jul 07 13:17:33.691134 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:33.690748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt"]
Jul 07 13:17:33.691886 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:33.691854    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8426c2_3e6c_43f1_950e_8ee0cfe3fcb1.slice/crio-b446967e6f77c0e1ddbb3b9b6a48af6bfc465b51cfbe9dc3a80d3cb66b582931 WatchSource:0}: Error finding container b446967e6f77c0e1ddbb3b9b6a48af6bfc465b51cfbe9dc3a80d3cb66b582931: Status 404 returned error can't find the container with id b446967e6f77c0e1ddbb3b9b6a48af6bfc465b51cfbe9dc3a80d3cb66b582931
Jul 07 13:17:33.695253 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:33.695233    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37a22f62_0d29_4c19_ba7b_4712ee5af95e.slice/crio-da915bdd5663687afe3b23dff0d9301f0c0fd7dff0944b48b262f960cccb1100 WatchSource:0}: Error finding container da915bdd5663687afe3b23dff0d9301f0c0fd7dff0944b48b262f960cccb1100: Status 404 returned error can't find the container with id da915bdd5663687afe3b23dff0d9301f0c0fd7dff0944b48b262f960cccb1100
Jul 07 13:17:34.179031 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.178992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-kztcl" event={"ID":"8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1","Type":"ContainerStarted","Data":"b446967e6f77c0e1ddbb3b9b6a48af6bfc465b51cfbe9dc3a80d3cb66b582931"}
Jul 07 13:17:34.180337 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.180314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj" event={"ID":"bc338116-d8e0-4bfd-a457-d2b7254994d2","Type":"ContainerStarted","Data":"3450f95cf5518e9d7c5f29d232e8376e23c64595476f01e2712e352272dfe9b8"}
Jul 07 13:17:34.180456 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.180352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj" event={"ID":"bc338116-d8e0-4bfd-a457-d2b7254994d2","Type":"ContainerStarted","Data":"fe4af6f71a195a5129adfab2a24ace795b781c7f64afac46ed30232e38200d84"}
Jul 07 13:17:34.180456 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.180385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:34.181371 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.181349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt" event={"ID":"37a22f62-0d29-4c19-ba7b-4712ee5af95e","Type":"ContainerStarted","Data":"da915bdd5663687afe3b23dff0d9301f0c0fd7dff0944b48b262f960cccb1100"}
Jul 07 13:17:34.183056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.183038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2fvnk" event={"ID":"4c9f9d2c-f1b8-41aa-af80-7854ce25f309","Type":"ContainerStarted","Data":"89e509581ed23e21dc5ffdb1f8c6de81aae3ab8f0d799019d23d344c7be51fec"}
Jul 07 13:17:34.206037 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.205997    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj" podStartSLOduration=2.205984834 podStartE2EDuration="2.205984834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:34.204790722 +0000 UTC m=+60.085162016" watchObservedRunningTime="2026-07-07 13:17:34.205984834 +0000 UTC m=+60.086356165"
Jul 07 13:17:34.226568 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.226519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2fvnk" podStartSLOduration=18.007296513 podStartE2EDuration="25.22650242s" totalImagesPullingTime="7.219205907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:17:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:34.225799866 +0000 UTC m=+60.106171184" watchObservedRunningTime="2026-07-07 13:17:34.22650242 +0000 UTC m=+60.106873715"
Jul 07 13:17:34.841175 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.841123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:34.940676 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:34.940640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:35.086898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:35.086870    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2pg9k"
Jul 07 13:17:35.202901 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:35.202825    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:35.304727 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:35.304693    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.019026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.018986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.109322 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.109284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.190259 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.190228    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"]
Jul 07 13:17:36.196438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.196408    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.245539 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.245502    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"]
Jul 07 13:17:36.245683 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.245620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:36.248033 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.247996    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 13:17:36.248201 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.248183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-96jwh\""
Jul 07 13:17:36.259049 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.258991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.311811 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.311752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.352572 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.352539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21ad6ceb-6375-49dc-893b-b7c6e2353305-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-79qth\" (UID: \"21ad6ceb-6375-49dc-893b-b7c6e2353305\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:36.453514 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.453486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21ad6ceb-6375-49dc-893b-b7c6e2353305-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-79qth\" (UID: \"21ad6ceb-6375-49dc-893b-b7c6e2353305\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:36.456390 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.456360    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21ad6ceb-6375-49dc-893b-b7c6e2353305-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-79qth\" (UID: \"21ad6ceb-6375-49dc-893b-b7c6e2353305\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:36.558345 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.558311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:36.596307 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.596267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.665971 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.665940    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:36.825814 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:36.825751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"]
Jul 07 13:17:36.831396 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:36.829310    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ad6ceb_6375_49dc_893b_b7c6e2353305.slice/crio-d99b5cc97e2925afdafbbf83f5956cff082ff77bd4d83bf753f2fb19ec274bac WatchSource:0}: Error finding container d99b5cc97e2925afdafbbf83f5956cff082ff77bd4d83bf753f2fb19ec274bac: Status 404 returned error can't find the container with id d99b5cc97e2925afdafbbf83f5956cff082ff77bd4d83bf753f2fb19ec274bac
Jul 07 13:17:37.202649 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:37.202278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt" event={"ID":"37a22f62-0d29-4c19-ba7b-4712ee5af95e","Type":"ContainerStarted","Data":"c14e2bb32dd4d68685dea1de76dd11ffdcd7d05baf796921ac976324681304ce"}
Jul 07 13:17:37.204094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:37.204070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth" event={"ID":"21ad6ceb-6375-49dc-893b-b7c6e2353305","Type":"ContainerStarted","Data":"d99b5cc97e2925afdafbbf83f5956cff082ff77bd4d83bf753f2fb19ec274bac"}
Jul 07 13:17:37.217649 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:37.217595    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d58d654b-2kdlt" podStartSLOduration=1.859448552 podStartE2EDuration="5.2175764s" totalImagesPullingTime="3.358127848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:37.216534453 +0000 UTC m=+63.096905760" watchObservedRunningTime="2026-07-07 13:17:37.2175764 +0000 UTC m=+63.097947695"
Jul 07 13:17:37.505845 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:37.505800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:37.566837 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:37.566794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:38.406138 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.406104    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-658fbb7f75-pv9sj"]
Jul 07 13:17:38.408513 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.408490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.410257 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.410232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:38.410920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.410895    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 13:17:38.411024 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.410925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 13:17:38.411808 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.411786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 13:17:38.411894 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.411864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 13:17:38.412044 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.412030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xh5f4\""
Jul 07 13:17:38.412194 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.412155    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 13:17:38.422661 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.422634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-658fbb7f75-pv9sj"]
Jul 07 13:17:38.468839 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.468812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:38.470971 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.470950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shkh\" (UniqueName: \"kubernetes.io/projected/438f1ee7-72d7-497a-bfa1-bc5f091014de-kube-api-access-4shkh\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.471083 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.470997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-service-ca\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.471135 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.471097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-oauth-config\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.471194 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.471156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-oauth-serving-cert\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.471233 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.471203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-config\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.471290 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.471263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-serving-cert\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572212 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-oauth-serving-cert\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572212 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-config\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-serving-cert\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shkh\" (UniqueName: \"kubernetes.io/projected/438f1ee7-72d7-497a-bfa1-bc5f091014de-kube-api-access-4shkh\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-service-ca\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-oauth-config\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.572982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572938    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-oauth-serving-cert\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.573197 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.573126    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-service-ca\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.573197 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.572946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-config\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.574985 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.574959    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-serving-cert\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.575078 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.575027    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-oauth-config\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.580514 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.580491    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shkh\" (UniqueName: \"kubernetes.io/projected/438f1ee7-72d7-497a-bfa1-bc5f091014de-kube-api-access-4shkh\") pod \"console-658fbb7f75-pv9sj\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") " pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.719276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.719235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:38.841811 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.841732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:38.860375 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.860348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-658fbb7f75-pv9sj"]
Jul 07 13:17:38.864108 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:38.864080    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438f1ee7_72d7_497a_bfa1_bc5f091014de.slice/crio-ec69c49d093c3b5473b32150bb9790a10f6c40a9fbf569133e7b8f53f63137c2 WatchSource:0}: Error finding container ec69c49d093c3b5473b32150bb9790a10f6c40a9fbf569133e7b8f53f63137c2: Status 404 returned error can't find the container with id ec69c49d093c3b5473b32150bb9790a10f6c40a9fbf569133e7b8f53f63137c2
Jul 07 13:17:38.949112 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:38.949082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:39.211407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.211322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658fbb7f75-pv9sj" event={"ID":"438f1ee7-72d7-497a-bfa1-bc5f091014de","Type":"ContainerStarted","Data":"ec69c49d093c3b5473b32150bb9790a10f6c40a9fbf569133e7b8f53f63137c2"}
Jul 07 13:17:39.212753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.212729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth" event={"ID":"21ad6ceb-6375-49dc-893b-b7c6e2353305","Type":"ContainerStarted","Data":"cf78f87999c4ba67fad79d02d05759965518605ba07f86caf8ca2f71e33a7094"}
Jul 07 13:17:39.212943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.212923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:39.218647 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.218605    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth"
Jul 07 13:17:39.228103 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.228056    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-79qth" podStartSLOduration=1.992639082 podStartE2EDuration="3.228042739s" totalImagesPullingTime="1.235403657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:39.227248 +0000 UTC m=+65.107619314" watchObservedRunningTime="2026-07-07 13:17:39.228042739 +0000 UTC m=+65.108414024"
Jul 07 13:17:39.479837 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.479752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:39.479837 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.479831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:39.482742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.482716    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 13:17:39.482742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.482738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 13:17:39.493358 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.493330    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81f506f0-9be6-48c3-b771-9e7648ada7a4-original-pull-secret\") pod \"global-pull-secret-syncer-mbpsm\" (UID: \"81f506f0-9be6-48c3-b771-9e7648ada7a4\") " pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:39.494751 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.494728    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/404f302b-7cf7-41ff-a9d1-a2ada86d1ab5-metrics-certs\") pod \"network-metrics-daemon-s6p2q\" (UID: \"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5\") " pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:39.580413 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.580382    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mbpsm"
Jul 07 13:17:39.580580 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.580551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:39.583527 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.583507    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 13:17:39.587886 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.587828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9nksx\""
Jul 07 13:17:39.592927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.592910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 13:17:39.596055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.596036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s6p2q"
Jul 07 13:17:39.606072 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.606047    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfgnp\" (UniqueName: \"kubernetes.io/projected/8f23e9b4-4c44-4420-aa9c-26e57abc4f7c-kube-api-access-wfgnp\") pod \"network-check-target-lxbcn\" (UID: \"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c\") " pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:39.739689 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.739608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mbpsm"]
Jul 07 13:17:39.744716 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:39.744686    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f506f0_9be6_48c3_b771_9e7648ada7a4.slice/crio-47b161fc273eed532d367bbea8947d8a7f7ad7d91db8aeb9f063d8d41bac8122 WatchSource:0}: Error finding container 47b161fc273eed532d367bbea8947d8a7f7ad7d91db8aeb9f063d8d41bac8122: Status 404 returned error can't find the container with id 47b161fc273eed532d367bbea8947d8a7f7ad7d91db8aeb9f063d8d41bac8122
Jul 07 13:17:39.763493 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.763469    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s6p2q"]
Jul 07 13:17:39.767086 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:39.766881    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404f302b_7cf7_41ff_a9d1_a2ada86d1ab5.slice/crio-bc481b53ca9d718ffbded866f9304d8345aecfc73708b50386187fa8eec58d25 WatchSource:0}: Error finding container bc481b53ca9d718ffbded866f9304d8345aecfc73708b50386187fa8eec58d25: Status 404 returned error can't find the container with id bc481b53ca9d718ffbded866f9304d8345aecfc73708b50386187fa8eec58d25
Jul 07 13:17:39.870230 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.870195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sgxmd\""
Jul 07 13:17:39.878317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:39.878292    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:40.017588 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.017550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lxbcn"]
Jul 07 13:17:40.022627 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:17:40.022596    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f23e9b4_4c44_4420_aa9c_26e57abc4f7c.slice/crio-de0c8560744d02e2a2f51fe6e543c9cb14bed6b16fcb4566f2caeecf1a9d5e4e WatchSource:0}: Error finding container de0c8560744d02e2a2f51fe6e543c9cb14bed6b16fcb4566f2caeecf1a9d5e4e: Status 404 returned error can't find the container with id de0c8560744d02e2a2f51fe6e543c9cb14bed6b16fcb4566f2caeecf1a9d5e4e
Jul 07 13:17:40.219771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.219642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s6p2q" event={"ID":"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5","Type":"ContainerStarted","Data":"bc481b53ca9d718ffbded866f9304d8345aecfc73708b50386187fa8eec58d25"}
Jul 07 13:17:40.220525 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.220457    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:40.221815 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.221739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbpsm" event={"ID":"81f506f0-9be6-48c3-b771-9e7648ada7a4","Type":"ContainerStarted","Data":"47b161fc273eed532d367bbea8947d8a7f7ad7d91db8aeb9f063d8d41bac8122"}
Jul 07 13:17:40.224356 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.224302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lxbcn" event={"ID":"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c","Type":"ContainerStarted","Data":"de0c8560744d02e2a2f51fe6e543c9cb14bed6b16fcb4566f2caeecf1a9d5e4e"}
Jul 07 13:17:40.256232 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.256207    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"]
Jul 07 13:17:40.258591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.258565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.261359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.261324    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 07 13:17:40.264262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.262627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sfcn9\""
Jul 07 13:17:40.264262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.262835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 13:17:40.264262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.263088    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 13:17:40.264262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.263274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 07 13:17:40.264262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.263450    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 13:17:40.272340 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.272300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"]
Jul 07 13:17:40.290288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.290152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:40.386405 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.386093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfd9v\" (UniqueName: \"kubernetes.io/projected/c7f94f31-2469-448f-8e9c-0180a0de52cc-kube-api-access-kfd9v\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.386405 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.386151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7f94f31-2469-448f-8e9c-0180a0de52cc-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.386405 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.386248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.386405 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.386289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.417199 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.417073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:40.488563 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.487502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfd9v\" (UniqueName: \"kubernetes.io/projected/c7f94f31-2469-448f-8e9c-0180a0de52cc-kube-api-access-kfd9v\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.488563 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.487550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7f94f31-2469-448f-8e9c-0180a0de52cc-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.488563 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.487621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.488563 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.487662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.488563 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:40.487790    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 07 13:17:40.488563 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:17:40.487856    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-tls podName:c7f94f31-2469-448f-8e9c-0180a0de52cc nodeName:}" failed. No retries permitted until 2026-07-07 13:17:40.987835528 +0000 UTC m=+66.868206805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-r8nms" (UID: "c7f94f31-2469-448f-8e9c-0180a0de52cc") : secret "prometheus-operator-tls" not found
Jul 07 13:17:40.489248 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.489019    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7f94f31-2469-448f-8e9c-0180a0de52cc-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.493493 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.493463    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.515774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.515739    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfd9v\" (UniqueName: \"kubernetes.io/projected/c7f94f31-2469-448f-8e9c-0180a0de52cc-kube-api-access-kfd9v\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.985143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.985058    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:40.991579 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.991551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:40.995085 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:40.995060    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7f94f31-2469-448f-8e9c-0180a0de52cc-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r8nms\" (UID: \"c7f94f31-2469-448f-8e9c-0180a0de52cc\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:41.174858 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:41.173969    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hbz7c"
Jul 07 13:17:41.179959 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:41.179756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"
Jul 07 13:17:41.366063 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:41.365980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms"]
Jul 07 13:17:42.240756 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:42.240712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s6p2q" event={"ID":"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5","Type":"ContainerStarted","Data":"89d38cbb4e2db3941bd6163cf7de5c930c115e08f3ea60ab491067644a6cb594"}
Jul 07 13:17:42.241232 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:42.240782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s6p2q" event={"ID":"404f302b-7cf7-41ff-a9d1-a2ada86d1ab5","Type":"ContainerStarted","Data":"543caa35ce1a463e4111418141e76180fc323d7bba2df5dde641a30ae33ff243"}
Jul 07 13:17:42.242889 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:42.242818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms" event={"ID":"c7f94f31-2469-448f-8e9c-0180a0de52cc","Type":"ContainerStarted","Data":"9118e79bd9f93b4d440fe8d0c16a5eb11f53cd5ddf53915884967723515695f4"}
Jul 07 13:17:42.351851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:42.351759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:42.415214 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:42.415151    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:43.198659 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:43.198622    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:46.999575 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:46.999525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:50.291076 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:50.291047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:50.347103 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:50.347048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:50.865420 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:50.865362    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s6p2q" podStartSLOduration=75.542837286 podStartE2EDuration="1m16.865335675s" totalImagesPullingTime="1.322498389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:42.256783088 +0000 UTC m=+68.137154400" watchObservedRunningTime="2026-07-07 13:17:50.865335675 +0000 UTC m=+76.745706969"
Jul 07 13:17:50.866557 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:50.866534    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-658fbb7f75-pv9sj"]
Jul 07 13:17:51.026756 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:51.026725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:51.090099 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:51.090066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:52.497717 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:52.497678    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:52.546282 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:52.546250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:53.598142 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:53.598103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:53.650445 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:53.650409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:54.819090 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:54.814941    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:54.911258 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:54.911181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:55.034003 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.033971    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:55.193973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.193906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bddc7bbc5-khlbj"
Jul 07 13:17:55.296291 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.296246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lxbcn" event={"ID":"8f23e9b4-4c44-4420-aa9c-26e57abc4f7c","Type":"ContainerStarted","Data":"bed605e8c75874de856fad1e082b388ebad5f7215a1eb2cff223078825fcdd14"}
Jul 07 13:17:55.296463 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.296364    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:17:55.298759 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.298729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms" event={"ID":"c7f94f31-2469-448f-8e9c-0180a0de52cc","Type":"ContainerStarted","Data":"36f120073d794b9a8cb72240035c6d3e64bea04e2ebbe760cc5ff13259a0e61e"}
Jul 07 13:17:55.298940 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.298798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms" event={"ID":"c7f94f31-2469-448f-8e9c-0180a0de52cc","Type":"ContainerStarted","Data":"0bf7d786d09d0c71e961ca4bb1454622281d291ec993e08fd639cb4122fb5b9f"}
Jul 07 13:17:55.300655 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.300622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-kztcl" event={"ID":"8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1","Type":"ContainerStarted","Data":"e5d9ec243b509c589cd5b7f9e938ba6c48d27c9f7e8a34308cef5835be5b4495"}
Jul 07 13:17:55.300770 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.300741    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:55.302866 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.302642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658fbb7f75-pv9sj" event={"ID":"438f1ee7-72d7-497a-bfa1-bc5f091014de","Type":"ContainerStarted","Data":"fe98811b89a8069ec5d2f369066e71326e84e132f88cee1225f688cf935136fe"}
Jul 07 13:17:55.304558 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.304533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mbpsm" event={"ID":"81f506f0-9be6-48c3-b771-9e7648ada7a4","Type":"ContainerStarted","Data":"5c985d4dae3dbd1bfae5d6c82688f745da71860ede683e88a71d77df218232f5"}
Jul 07 13:17:55.310732 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.310655    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lxbcn" podStartSLOduration=66.70420817 podStartE2EDuration="1m21.310638177s" totalImagesPullingTime="14.606430007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:55.310352507 +0000 UTC m=+81.190723818" watchObservedRunningTime="2026-07-07 13:17:55.310638177 +0000 UTC m=+81.191009472"
Jul 07 13:17:55.314575 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.314551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:55.317571 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.317379    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-kztcl"
Jul 07 13:17:55.325524 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.325477    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mbpsm" podStartSLOduration=66.44685778 podStartE2EDuration="1m21.325461166s" totalImagesPullingTime="14.878603386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:55.325009469 +0000 UTC m=+81.205380764" watchObservedRunningTime="2026-07-07 13:17:55.325461166 +0000 UTC m=+81.205832460"
Jul 07 13:17:55.342361 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.342317    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-658fbb7f75-pv9sj" podStartSLOduration=1.576857225 podStartE2EDuration="17.342299843s" totalImagesPullingTime="15.765442618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:55.34079496 +0000 UTC m=+81.221166281" watchObservedRunningTime="2026-07-07 13:17:55.342299843 +0000 UTC m=+81.222671144"
Jul 07 13:17:55.357101 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.357041    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r8nms" podStartSLOduration=2.095678349 podStartE2EDuration="15.357028107s" totalImagesPullingTime="13.261349758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:55.356382098 +0000 UTC m=+81.236753422" watchObservedRunningTime="2026-07-07 13:17:55.357028107 +0000 UTC m=+81.237399401"
Jul 07 13:17:55.371758 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.371718    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-kztcl" podStartSLOduration=2.377268861 podStartE2EDuration="23.371706706s" totalImagesPullingTime="20.994437845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:17:55.370829308 +0000 UTC m=+81.251200632" watchObservedRunningTime="2026-07-07 13:17:55.371706706 +0000 UTC m=+81.252078001"
Jul 07 13:17:55.378404 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:55.378366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:57.332955 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:57.332917    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:57.800328 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:57.800286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:57.871913 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:57.871866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:57.941055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:57.940992    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:58.720100 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:58.720063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:17:59.312253 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:59.312185    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:17:59.369334 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:17:59.369286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:00.112388 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:00.112328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:00.170377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:00.170345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:01.439912 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.439873    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:01.465535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.465501    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"]
Jul 07 13:18:01.499148 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.499115    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"]
Jul 07 13:18:01.499314 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.499262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.502368 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.502341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 13:18:01.502513 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.502376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m8d4v\""
Jul 07 13:18:01.502598 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.502356    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 13:18:01.583429 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.583391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:01.588306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.588278    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"]
Jul 07 13:18:01.609566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.609540    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nszwn"]
Jul 07 13:18:01.609714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.609662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.612397 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.612373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 07 13:18:01.612600 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.612540    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zhlq8\""
Jul 07 13:18:01.612755 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.612608    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 07 13:18:01.612827 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.612689    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 07 13:18:01.636241 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.636210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"]
Jul 07 13:18:01.636400 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.636384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.639463 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.639437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 13:18:01.639463 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.639455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 13:18:01.639621 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.639527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 13:18:01.639621 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.639460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wjwb8\""
Jul 07 13:18:01.659672 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.659641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.659767 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.659689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.659767 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.659725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.659767 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.659758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsxz\" (UniqueName: \"kubernetes.io/projected/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-kube-api-access-xfsxz\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.662681 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.662656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:01.761214 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/23094b01-90f4-46b1-9940-05bd7c0fbe16-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-textfile\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10691b1d-77e6-414f-b4a4-d204de1be760-metrics-client-ca\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23094b01-90f4-46b1-9940-05bd7c0fbe16-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-root\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsxz\" (UniqueName: \"kubernetes.io/projected/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-kube-api-access-xfsxz\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24n56\" (UniqueName: \"kubernetes.io/projected/10691b1d-77e6-414f-b4a4-d204de1be760-kube-api-access-24n56\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-tls\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-wtmp\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr55v\" (UniqueName: \"kubernetes.io/projected/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-api-access-hr55v\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-sys\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.761793 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.761749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-accelerators-collector-config\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.762306 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:18:01.761963    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 13:18:01.762381 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:18:01.762362    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-tls podName:efecd0e8-7e87-47d1-aa13-5f7e2815adf0 nodeName:}" failed. No retries permitted until 2026-07-07 13:18:02.262336746 +0000 UTC m=+88.142708021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-g9z2m" (UID: "efecd0e8-7e87-47d1-aa13-5f7e2815adf0") : secret "openshift-state-metrics-tls" not found
Jul 07 13:18:01.762782 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.762740    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.765254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.765231    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.773555 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.773530    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsxz\" (UniqueName: \"kubernetes.io/projected/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-kube-api-access-xfsxz\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:01.829765 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.829728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:01.862769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.862769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24n56\" (UniqueName: \"kubernetes.io/projected/10691b1d-77e6-414f-b4a4-d204de1be760-kube-api-access-24n56\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-tls\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-wtmp\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr55v\" (UniqueName: \"kubernetes.io/projected/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-api-access-hr55v\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-sys\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-accelerators-collector-config\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/23094b01-90f4-46b1-9940-05bd7c0fbe16-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.862983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.863013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.863448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-textfile\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10691b1d-77e6-414f-b4a4-d204de1be760-metrics-client-ca\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23094b01-90f4-46b1-9940-05bd7c0fbe16-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.863448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-root\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863261    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-root\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.863751 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863570    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/23094b01-90f4-46b1-9940-05bd7c0fbe16-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.863803 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.863782    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-accelerators-collector-config\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.864210 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.864180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.864339 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.864233    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-wtmp\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.864410 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:18:01.864343    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 13:18:01.864410 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:18:01.864406    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-tls podName:10691b1d-77e6-414f-b4a4-d204de1be760 nodeName:}" failed. No retries permitted until 2026-07-07 13:18:02.364387709 +0000 UTC m=+88.244758994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-tls") pod "node-exporter-nszwn" (UID: "10691b1d-77e6-414f-b4a4-d204de1be760") : secret "node-exporter-tls" not found
Jul 07 13:18:01.865115 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.865087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10691b1d-77e6-414f-b4a4-d204de1be760-metrics-client-ca\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.865261 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.865154    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23094b01-90f4-46b1-9940-05bd7c0fbe16-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.865261 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.865180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10691b1d-77e6-414f-b4a4-d204de1be760-sys\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.865400 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.865314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-textfile\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.868687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.868643    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.869717 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.869695    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.869857 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.869832    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.874946 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.874922    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24n56\" (UniqueName: \"kubernetes.io/projected/10691b1d-77e6-414f-b4a4-d204de1be760-kube-api-access-24n56\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:01.876825 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.876500    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr55v\" (UniqueName: \"kubernetes.io/projected/23094b01-90f4-46b1-9940-05bd7c0fbe16-kube-api-access-hr55v\") pod \"kube-state-metrics-7c9dbb8b7d-krdlm\" (UID: \"23094b01-90f4-46b1-9940-05bd7c0fbe16\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.923339 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.923300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"
Jul 07 13:18:01.928684 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:01.928558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:02.033239 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.032973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:02.129186 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.125842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm"]
Jul 07 13:18:02.129186 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.128457    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:02.130389 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:18:02.130359    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23094b01_90f4_46b1_9940_05bd7c0fbe16.slice/crio-c67df14b2f53ffa4672be05865b620ad365a739fc6a6af28cb3a81f122f7735e WatchSource:0}: Error finding container c67df14b2f53ffa4672be05865b620ad365a739fc6a6af28cb3a81f122f7735e: Status 404 returned error can't find the container with id c67df14b2f53ffa4672be05865b620ad365a739fc6a6af28cb3a81f122f7735e
Jul 07 13:18:02.181960 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.181930    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:02.266494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.266450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:02.269323 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.269298    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efecd0e8-7e87-47d1-aa13-5f7e2815adf0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-g9z2m\" (UID: \"efecd0e8-7e87-47d1-aa13-5f7e2815adf0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:02.333480 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.333396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm" event={"ID":"23094b01-90f4-46b1-9940-05bd7c0fbe16","Type":"ContainerStarted","Data":"c67df14b2f53ffa4672be05865b620ad365a739fc6a6af28cb3a81f122f7735e"}
Jul 07 13:18:02.367945 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.367911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-tls\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:02.370753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.370725    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10691b1d-77e6-414f-b4a4-d204de1be760-node-exporter-tls\") pod \"node-exporter-nszwn\" (UID: \"10691b1d-77e6-414f-b4a4-d204de1be760\") " pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:02.411834 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.411807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"
Jul 07 13:18:02.548761 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.548219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nszwn"
Jul 07 13:18:02.558198 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.558070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:02.584390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:18:02.584318    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10691b1d_77e6_414f_b4a4_d204de1be760.slice/crio-6cf70c9800d8cfadbe3d8db6eacf2ed55d375ea33c06754f098551520e189190 WatchSource:0}: Error finding container 6cf70c9800d8cfadbe3d8db6eacf2ed55d375ea33c06754f098551520e189190: Status 404 returned error can't find the container with id 6cf70c9800d8cfadbe3d8db6eacf2ed55d375ea33c06754f098551520e189190
Jul 07 13:18:02.604936 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.604908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m"]
Jul 07 13:18:02.610857 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:18:02.610822    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefecd0e8_7e87_47d1_aa13_5f7e2815adf0.slice/crio-9f49ce8b47e0feb2789a0fb874cf989b203421b3495099e52c08d4ff8908006d WatchSource:0}: Error finding container 9f49ce8b47e0feb2789a0fb874cf989b203421b3495099e52c08d4ff8908006d: Status 404 returned error can't find the container with id 9f49ce8b47e0feb2789a0fb874cf989b203421b3495099e52c08d4ff8908006d
Jul 07 13:18:02.654566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:02.654521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:03.343728 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.343617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszwn" event={"ID":"10691b1d-77e6-414f-b4a4-d204de1be760","Type":"ContainerStarted","Data":"6cf70c9800d8cfadbe3d8db6eacf2ed55d375ea33c06754f098551520e189190"}
Jul 07 13:18:03.346522 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.346488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m" event={"ID":"efecd0e8-7e87-47d1-aa13-5f7e2815adf0","Type":"ContainerStarted","Data":"19eba5278876431db8a846840cdad0cb9b317c99151895f44a2b9a3783c07adb"}
Jul 07 13:18:03.346654 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.346554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m" event={"ID":"efecd0e8-7e87-47d1-aa13-5f7e2815adf0","Type":"ContainerStarted","Data":"7cae500d5fd4b42411613e7ded27ac6826191788bc774750be9362e7b3b794c4"}
Jul 07 13:18:03.346654 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.346583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m" event={"ID":"efecd0e8-7e87-47d1-aa13-5f7e2815adf0","Type":"ContainerStarted","Data":"9f49ce8b47e0feb2789a0fb874cf989b203421b3495099e52c08d4ff8908006d"}
Jul 07 13:18:03.562318 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.560328    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"]
Jul 07 13:18:03.595551 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.595070    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"]
Jul 07 13:18:03.595551 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.595261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.598553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.598524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 13:18:03.599313 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.599189    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 13:18:03.599424 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.599395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3a8gu7a3sin97\""
Jul 07 13:18:03.600059 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.599632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zq296\""
Jul 07 13:18:03.600059 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.599819    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 13:18:03.600059 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.599919    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 13:18:03.600059 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.599970    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 13:18:03.683030 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.682754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683030 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.682818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683030 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.682847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-metrics-client-ca\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683030 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.682953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-tls\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683030 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.682980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683512 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.683059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-grpc-tls\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683512 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.683141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dbfc\" (UniqueName: \"kubernetes.io/projected/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-kube-api-access-9dbfc\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.683512 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.683248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.783961 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.783916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.783970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-metrics-client-ca\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.784001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-tls\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.784027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.784056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-grpc-tls\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.784095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dbfc\" (UniqueName: \"kubernetes.io/projected/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-kube-api-access-9dbfc\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.784122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.784508 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.784191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.785631 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.785601    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-metrics-client-ca\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.791839 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.791812    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.793860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.793830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.802437 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.802380    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.807507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.806815    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dbfc\" (UniqueName: \"kubernetes.io/projected/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-kube-api-access-9dbfc\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.807507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.807245    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.807507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.807314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-thanos-querier-tls\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.807507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.807412    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/259a1bfa-4469-4ad4-8742-35e0ae7f1da8-secret-grpc-tls\") pod \"thanos-querier-84f6f8f7f6-tbwx5\" (UID: \"259a1bfa-4469-4ad4-8742-35e0ae7f1da8\") " pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:03.891828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.891734    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:03.911150 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:03.910885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:04.510494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:04.510461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:04.611125 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:04.611090    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:04.713845 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:04.713812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:05.098422 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.098383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:05.425276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.425146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"]
Jul 07 13:18:05.428769 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:18:05.428743    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259a1bfa_4469_4ad4_8742_35e0ae7f1da8.slice/crio-4d4cfa63272fa3968a6344a3aacf77d62bbecf16960e8263b7ab154692f03a93 WatchSource:0}: Error finding container 4d4cfa63272fa3968a6344a3aacf77d62bbecf16960e8263b7ab154692f03a93: Status 404 returned error can't find the container with id 4d4cfa63272fa3968a6344a3aacf77d62bbecf16960e8263b7ab154692f03a93
Jul 07 13:18:05.919292 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.919213    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64db89fcd9-jdqlw"]
Jul 07 13:18:05.932394 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.932365    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64db89fcd9-jdqlw"]
Jul 07 13:18:05.932551 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.932481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:05.935282 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.935263    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 13:18:05.936412 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.936385    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 13:18:05.936412 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.936395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-28vevlcagmufq\""
Jul 07 13:18:05.936579 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.936452    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 13:18:05.936579 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.936397    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 13:18:05.936579 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:05.936570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fkpln\""
Jul 07 13:18:06.107347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-metrics-server-audit-profiles\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.107347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-audit-log\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.107541 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-secret-metrics-server-tls\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.107541 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-secret-metrics-server-client-certs\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.107541 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-client-ca-bundle\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.107541 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.107663 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.107577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm2w\" (UniqueName: \"kubernetes.io/projected/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-kube-api-access-fdm2w\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208403 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208403 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm2w\" (UniqueName: \"kubernetes.io/projected/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-kube-api-access-fdm2w\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-metrics-server-audit-profiles\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-audit-log\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-secret-metrics-server-tls\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-secret-metrics-server-client-certs\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-client-ca-bundle\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.208873 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.208853    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-audit-log\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.209142 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.209111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.209450 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.209409    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-metrics-server-audit-profiles\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.211180 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.211142    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-client-ca-bundle\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.211279 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.211188    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-secret-metrics-server-tls\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.211317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.211280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-secret-metrics-server-client-certs\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.216450 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.216431    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm2w\" (UniqueName: \"kubernetes.io/projected/1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b-kube-api-access-fdm2w\") pod \"metrics-server-64db89fcd9-jdqlw\" (UID: \"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b\") " pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.244414 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.244389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:06.361786 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.361753    2581 generic.go:356] "Generic (PLEG): container finished" podID="10691b1d-77e6-414f-b4a4-d204de1be760" containerID="17ce4ac27090759a28cb2fc77633e6f38f857a8faf9a6565573952e24c7d296d" exitCode=0
Jul 07 13:18:06.361964 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.361847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszwn" event={"ID":"10691b1d-77e6-414f-b4a4-d204de1be760","Type":"ContainerDied","Data":"17ce4ac27090759a28cb2fc77633e6f38f857a8faf9a6565573952e24c7d296d"}
Jul 07 13:18:06.364860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.364320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm" event={"ID":"23094b01-90f4-46b1-9940-05bd7c0fbe16","Type":"ContainerStarted","Data":"58b2a27d50f53eb0892b083b0c829fd9f4d793fc49585904d9e2437d6f9f682a"}
Jul 07 13:18:06.364860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.364373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm" event={"ID":"23094b01-90f4-46b1-9940-05bd7c0fbe16","Type":"ContainerStarted","Data":"36c5289e01f5638a52cb0ee30f255f43de0becbc1cc2377f6ca323547a6f4d44"}
Jul 07 13:18:06.364860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.364400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm" event={"ID":"23094b01-90f4-46b1-9940-05bd7c0fbe16","Type":"ContainerStarted","Data":"df5d88873f6d1363428eb8c2872b4bf65b30510c1c049114c4736ce7bbdba879"}
Jul 07 13:18:06.367055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.367031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m" event={"ID":"efecd0e8-7e87-47d1-aa13-5f7e2815adf0","Type":"ContainerStarted","Data":"3d1e62d52d58e117b765ef23da37b9d34a433b1c9857697d2f0a7e14ee073637"}
Jul 07 13:18:06.368629 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.368582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"4d4cfa63272fa3968a6344a3aacf77d62bbecf16960e8263b7ab154692f03a93"}
Jul 07 13:18:06.373092 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.373039    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:06.417118 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.415984    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-g9z2m" podStartSLOduration=3.009838635 podStartE2EDuration="5.415963804s" totalImagesPullingTime="2.406125169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:18:06.414539078 +0000 UTC m=+92.294910393" watchObservedRunningTime="2026-07-07 13:18:06.415963804 +0000 UTC m=+92.296335097"
Jul 07 13:18:06.428967 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.428940    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64db89fcd9-jdqlw"]
Jul 07 13:18:06.433101 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:18:06.433069    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a039c05_d8f5_4f2d_9dc4_f1d9c94d6d5b.slice/crio-30667f35d341c11d9537396bdd78c58d575e720bd4db3a75bd1fa6305e6bc6ab WatchSource:0}: Error finding container 30667f35d341c11d9537396bdd78c58d575e720bd4db3a75bd1fa6305e6bc6ab: Status 404 returned error can't find the container with id 30667f35d341c11d9537396bdd78c58d575e720bd4db3a75bd1fa6305e6bc6ab
Jul 07 13:18:06.435857 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.435789    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-krdlm" podStartSLOduration=2.285829177 podStartE2EDuration="5.435769185s" totalImagesPullingTime="3.149940008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:18:06.435152081 +0000 UTC m=+92.315523405" watchObservedRunningTime="2026-07-07 13:18:06.435769185 +0000 UTC m=+92.316140480"
Jul 07 13:18:06.461681 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.461619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:06.562089 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.562043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:06.662559 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.662527    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:06.771185 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:06.771126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:07.376508 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.376449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszwn" event={"ID":"10691b1d-77e6-414f-b4a4-d204de1be760","Type":"ContainerStarted","Data":"a8ccfc72f5a2bb89cb58cbde244824c0c14b072ef40cb447f254d3f7e6cb7026"}
Jul 07 13:18:07.377041 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.376571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszwn" event={"ID":"10691b1d-77e6-414f-b4a4-d204de1be760","Type":"ContainerStarted","Data":"2455e92251296abb14db898d6b5411f800697197bf8b34b3d2e5299d56c9b20c"}
Jul 07 13:18:07.378300 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.378266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw" event={"ID":"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b","Type":"ContainerStarted","Data":"30667f35d341c11d9537396bdd78c58d575e720bd4db3a75bd1fa6305e6bc6ab"}
Jul 07 13:18:07.395900 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.395849    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nszwn" podStartSLOduration=3.592342899 podStartE2EDuration="6.395833771s" totalImagesPullingTime="2.698123401s" totalInitContainerRuntime="105.367471ms" isStatefulPod=true podCreationTimestamp="2026-07-07 13:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:18:07.394513301 +0000 UTC m=+93.274884613" watchObservedRunningTime="2026-07-07 13:18:07.395833771 +0000 UTC m=+93.276205064"
Jul 07 13:18:07.505033 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.503153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:07.606647 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.606591    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:07.777722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.777684    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 13:18:07.778973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.778948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:07.798712 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.798681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 13:18:07.798873 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.798850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.801679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.801971    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.802326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nfrpp\""
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.802563    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.802781    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-556o9ucl6kgdp\""
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.803010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 13:18:07.803458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.803201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 13:18:07.803893 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.803715    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 13:18:07.803947 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.803906    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 13:18:07.803947 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.803935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 13:18:07.804521 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.804108    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 13:18:07.804521 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.804139    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 13:18:07.804521 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.804395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 13:18:07.805780 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.805645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 13:18:07.807080 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.807055    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 13:18:07.810810 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.810789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 13:18:07.923646 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.923646 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.923863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.923863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.923863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-config\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.923863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62596082-a023-4eab-b3ff-161ef7fd16fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.923863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.923999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssh5j\" (UniqueName: \"kubernetes.io/projected/62596082-a023-4eab-b3ff-161ef7fd16fd-kube-api-access-ssh5j\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62596082-a023-4eab-b3ff-161ef7fd16fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:07.924465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:07.924391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026132 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-config\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62596082-a023-4eab-b3ff-161ef7fd16fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026312 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.026719 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.026434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.027921 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.027851    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.028458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.028431    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.029078 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.029048    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.029581 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.029553    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.029860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.029838    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030020 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.029993    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030280 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030237    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-config\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030385 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030451 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030378    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030451 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssh5j\" (UniqueName: \"kubernetes.io/projected/62596082-a023-4eab-b3ff-161ef7fd16fd-kube-api-access-ssh5j\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62596082-a023-4eab-b3ff-161ef7fd16fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030707 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.030707 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.030599    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.031077 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.031050    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.032287 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.031692    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.032287 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.032091    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.032553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.032506    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/62596082-a023-4eab-b3ff-161ef7fd16fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.035503 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.035456    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/62596082-a023-4eab-b3ff-161ef7fd16fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.036512 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.035676    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.036512 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.036453    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/62596082-a023-4eab-b3ff-161ef7fd16fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.037527 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.037505    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.038731 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.038709    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.039899 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.039857    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/62596082-a023-4eab-b3ff-161ef7fd16fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.041306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.041259    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssh5j\" (UniqueName: \"kubernetes.io/projected/62596082-a023-4eab-b3ff-161ef7fd16fd-kube-api-access-ssh5j\") pod \"prometheus-k8s-0\" (UID: \"62596082-a023-4eab-b3ff-161ef7fd16fd\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.114041 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.113768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:08.394434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.394309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:08.495357 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:08.495314    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:09.285687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:09.285546    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 13:18:09.390119 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:09.390087    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"20bbeb28b0311da5258322475f597c6903d2d23ecae6c059c64c8316fdef35ff"}
Jul 07 13:18:09.391671 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:09.391648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw" event={"ID":"1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b","Type":"ContainerStarted","Data":"ed65922810edf7df91e6803964c3395e326f46e47d9693b326361e71401fc884"}
Jul 07 13:18:09.392858 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:09.392834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"84c9e5bb7e4c779d380fa26b0cca201ed5f6ee5c311878ce439401b724c88e91"}
Jul 07 13:18:09.415413 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:09.415362    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw" podStartSLOduration=1.711813755 podStartE2EDuration="4.41534338s" totalImagesPullingTime="2.703529625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:18:09.414232542 +0000 UTC m=+95.294603863" watchObservedRunningTime="2026-07-07 13:18:09.41534338 +0000 UTC m=+95.295714679"
Jul 07 13:18:10.399313 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:10.399282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"e1b7877690c39d957aa7ab9fc1c599058216f20a2bb74d91deabf8c5c6e15ded"}
Jul 07 13:18:10.399582 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:10.399346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"e291f3907f638bd609992806cb0d19d4756545a0258c06edc7332949aec8a33b"}
Jul 07 13:18:11.405076 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:11.405043    2581 generic.go:356] "Generic (PLEG): container finished" podID="62596082-a023-4eab-b3ff-161ef7fd16fd" containerID="98185bb470997cf1dc34d6ab0f3f8c7e0f7ba6f9ce35e107f199d02bbaf10c3a" exitCode=0
Jul 07 13:18:11.405501 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:11.405133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerDied","Data":"98185bb470997cf1dc34d6ab0f3f8c7e0f7ba6f9ce35e107f199d02bbaf10c3a"}
Jul 07 13:18:11.408023 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:11.408003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"e5ec257535080edc797ad560942e12fd859b1143c668673ed244c127bed78be3"}
Jul 07 13:18:11.408154 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:11.408056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"126e1f37bc195506a04d8cd23bb249121bae344bab7a734199215018f963c738"}
Jul 07 13:18:12.416552 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:12.416515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" event={"ID":"259a1bfa-4469-4ad4-8742-35e0ae7f1da8","Type":"ContainerStarted","Data":"2fd095b5a4f9b040c9cf4bf91a3b53e360e027d6e22acca03017b880d7266593"}
Jul 07 13:18:12.417473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:12.417445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:12.441706 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:12.441657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5" podStartSLOduration=4.218238496 podStartE2EDuration="9.441644247s" totalImagesPullingTime="5.223405751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:18:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:18:12.440618738 +0000 UTC m=+98.320990058" watchObservedRunningTime="2026-07-07 13:18:12.441644247 +0000 UTC m=+98.322015542"
Jul 07 13:18:13.984658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:13.984580    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnwlb_bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2/serve-healthcheck-canary/0.log"
Jul 07 13:18:14.434407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:14.434381    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84f6f8f7f6-tbwx5"
Jul 07 13:18:16.438373 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.438343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"8d17b4848f05cbb97bd0e1edf1d64eb48ada710f98798efd1b2e125c6a742bfc"}
Jul 07 13:18:16.438871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.438416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"a287866de6fd537c6508969a145a0aee397cb812012950f5cfa09094f17b4a9e"}
Jul 07 13:18:16.438871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.438468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"143a447f37e8cbbd727c5f7a1c910eb8996e9c576d11f713be28bcb43d50a64b"}
Jul 07 13:18:16.438871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.438511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"77a8ffb9e891c5c13e6c59badc766e993f077b61a4b3ed86d75b31b0b7450d41"}
Jul 07 13:18:16.438871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.438560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"1fdd5662583554c84e18afc16df33041007beb74055cc53f9bd06e28d4706d18"}
Jul 07 13:18:16.438871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.438599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"62596082-a023-4eab-b3ff-161ef7fd16fd","Type":"ContainerStarted","Data":"25dfcb009ec7ea33e16db1be91b548f790a9a5ced4f0768a235a30559d5a59c1"}
Jul 07 13:18:16.466537 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:16.466492    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.745105252 podStartE2EDuration="9.466477104s" totalImagesPullingTime="5.695652525s" totalInitContainerRuntime="25.719327ms" isStatefulPod=false podCreationTimestamp="2026-07-07 13:18:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:18:16.46461963 +0000 UTC m=+102.344990953" watchObservedRunningTime="2026-07-07 13:18:16.466477104 +0000 UTC m=+102.346848398"
Jul 07 13:18:17.131974 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:17.131939    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:17.183323 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:17.183296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:17.236419 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:17.236392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:18.113891 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:18.113859    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:18:20.328208 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.328151    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-658fbb7f75-pv9sj" podUID="438f1ee7-72d7-497a-bfa1-bc5f091014de" containerName="console" containerID="cri-o://fe98811b89a8069ec5d2f369066e71326e84e132f88cee1225f688cf935136fe" gracePeriod=15
Jul 07 13:18:20.455791 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.455761    2581 generic.go:356] "Generic (PLEG): container finished" podID="438f1ee7-72d7-497a-bfa1-bc5f091014de" containerID="fe98811b89a8069ec5d2f369066e71326e84e132f88cee1225f688cf935136fe" exitCode=0
Jul 07 13:18:20.455913 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.455821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658fbb7f75-pv9sj" event={"ID":"438f1ee7-72d7-497a-bfa1-bc5f091014de","Type":"ContainerDied","Data":"fe98811b89a8069ec5d2f369066e71326e84e132f88cee1225f688cf935136fe"}
Jul 07 13:18:20.623841 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.623809    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:18:20.652696 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.652675    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-service-ca\") pod \"438f1ee7-72d7-497a-bfa1-bc5f091014de\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") "
Jul 07 13:18:20.652848 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.652709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-config\") pod \"438f1ee7-72d7-497a-bfa1-bc5f091014de\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") "
Jul 07 13:18:20.652848 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.652764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-serving-cert\") pod \"438f1ee7-72d7-497a-bfa1-bc5f091014de\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") "
Jul 07 13:18:20.652848 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.652803    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-oauth-config\") pod \"438f1ee7-72d7-497a-bfa1-bc5f091014de\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") "
Jul 07 13:18:20.653026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.652937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-oauth-serving-cert\") pod \"438f1ee7-72d7-497a-bfa1-bc5f091014de\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") "
Jul 07 13:18:20.653085 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.653022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4shkh\" (UniqueName: \"kubernetes.io/projected/438f1ee7-72d7-497a-bfa1-bc5f091014de-kube-api-access-4shkh\") pod \"438f1ee7-72d7-497a-bfa1-bc5f091014de\" (UID: \"438f1ee7-72d7-497a-bfa1-bc5f091014de\") "
Jul 07 13:18:20.653139 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.653079    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-service-ca" (OuterVolumeSpecName: "service-ca") pod "438f1ee7-72d7-497a-bfa1-bc5f091014de" (UID: "438f1ee7-72d7-497a-bfa1-bc5f091014de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:18:20.653231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.653195    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-config" (OuterVolumeSpecName: "console-config") pod "438f1ee7-72d7-497a-bfa1-bc5f091014de" (UID: "438f1ee7-72d7-497a-bfa1-bc5f091014de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:18:20.653331 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.653311    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "438f1ee7-72d7-497a-bfa1-bc5f091014de" (UID: "438f1ee7-72d7-497a-bfa1-bc5f091014de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:18:20.653390 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.653338    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:18:20.653390 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.653353    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-config\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:18:20.655373 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.655345    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "438f1ee7-72d7-497a-bfa1-bc5f091014de" (UID: "438f1ee7-72d7-497a-bfa1-bc5f091014de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:18:20.655373 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.655366    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438f1ee7-72d7-497a-bfa1-bc5f091014de-kube-api-access-4shkh" (OuterVolumeSpecName: "kube-api-access-4shkh") pod "438f1ee7-72d7-497a-bfa1-bc5f091014de" (UID: "438f1ee7-72d7-497a-bfa1-bc5f091014de"). InnerVolumeSpecName "kube-api-access-4shkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:18:20.655511 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.655396    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "438f1ee7-72d7-497a-bfa1-bc5f091014de" (UID: "438f1ee7-72d7-497a-bfa1-bc5f091014de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:18:20.754755 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.754729    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4shkh\" (UniqueName: \"kubernetes.io/projected/438f1ee7-72d7-497a-bfa1-bc5f091014de-kube-api-access-4shkh\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:18:20.754755 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.754751    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-serving-cert\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:18:20.754755 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.754759    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/438f1ee7-72d7-497a-bfa1-bc5f091014de-console-oauth-config\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:18:20.754995 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:20.754768    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/438f1ee7-72d7-497a-bfa1-bc5f091014de-oauth-serving-cert\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:18:21.461449 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:21.461418    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-658fbb7f75-pv9sj"
Jul 07 13:18:21.461449 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:21.461432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-658fbb7f75-pv9sj" event={"ID":"438f1ee7-72d7-497a-bfa1-bc5f091014de","Type":"ContainerDied","Data":"ec69c49d093c3b5473b32150bb9790a10f6c40a9fbf569133e7b8f53f63137c2"}
Jul 07 13:18:21.462041 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:21.461506    2581 scope.go:117] "RemoveContainer" containerID="fe98811b89a8069ec5d2f369066e71326e84e132f88cee1225f688cf935136fe"
Jul 07 13:18:21.477601 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:21.477581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-658fbb7f75-pv9sj"]
Jul 07 13:18:21.481069 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:21.481046    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-658fbb7f75-pv9sj"]
Jul 07 13:18:22.492650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:22.492619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:22.550661 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:22.550632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:22.759241 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:22.759152    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438f1ee7-72d7-497a-bfa1-bc5f091014de" path="/var/lib/kubelet/pods/438f1ee7-72d7-497a-bfa1-bc5f091014de/volumes"
Jul 07 13:18:26.245529 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:26.245492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:26.246045 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:26.245564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:26.313695 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:26.312814    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lxbcn"
Jul 07 13:18:29.569058 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:29.569028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:29.672874 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:29.672850    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:29.729574 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:29.729545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:42.272987 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:42.272949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:18:46.250361 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:46.250330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:46.254107 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:46.254082    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64db89fcd9-jdqlw"
Jul 07 13:18:50.998732 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:18:50.998697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:08.114640 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:08.114609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:19:08.134797 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:08.134769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:19:08.669148 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:08.669120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 13:19:20.759709 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:20.759676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:21.151272 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:21.151152    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:30.000900 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:30.000811    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:35.928218 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:35.928183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:36.032722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:36.032690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:36.179122 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:36.179055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:44.056477 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:44.056439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:44.153530 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:44.153503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:47.908221 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:47.908180    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:47.967574 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:47.967539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:51.502386 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:51.502349    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:51.554839 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:51.554805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:51.609772 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:51.609739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:52.500106 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:52.500069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:19:52.600891 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:19:52.600865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:19.383572 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:19.383535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:24.999147 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:24.999109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:25.093715 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:25.093690    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:39.431845 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:39.431810    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:51.677254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:51.677211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:52.469404 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:52.469369    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:55.281804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:55.281730    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:55.382867 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:55.382823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:56.041816 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:56.041780    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:56.151657 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:56.151629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:56.266066 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:56.266038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:58.400857 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:58.400821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:20:58.790976 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:20:58.790943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:01.876829 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:01.876800    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:01.980920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:01.980882    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:05.595148 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:05.595112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:09.159650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:09.159616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:09.263892 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:09.263859    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:14.513082 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:14.513048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:14.613279 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:14.613249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:16.845557 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:16.845523    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:22.037584 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:22.037534    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:22.138203 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:22.138155    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:27.086584 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.086551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:27.125143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.125111    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"]
Jul 07 13:21:27.125483 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.125469    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438f1ee7-72d7-497a-bfa1-bc5f091014de" containerName="console"
Jul 07 13:21:27.125527 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.125486    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="438f1ee7-72d7-497a-bfa1-bc5f091014de" containerName="console"
Jul 07 13:21:27.125561 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.125538    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="438f1ee7-72d7-497a-bfa1-bc5f091014de" containerName="console"
Jul 07 13:21:27.127555 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.127536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.130731 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.130706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 13:21:27.131650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.131634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 13:21:27.131772 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.131738    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 13:21:27.131877 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.131852    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 13:21:27.131930 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.131919    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 13:21:27.132134 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.132113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kl9xm\""
Jul 07 13:21:27.141655 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.141634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"]
Jul 07 13:21:27.209605 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.209572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29pt\" (UniqueName: \"kubernetes.io/projected/c6963abc-7177-4904-ae7b-68867e0eb627-kube-api-access-m29pt\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.209746 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.209663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6963abc-7177-4904-ae7b-68867e0eb627-metrics-cert\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.209746 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.209714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c6963abc-7177-4904-ae7b-68867e0eb627-manager-config\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.209854 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.209750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6963abc-7177-4904-ae7b-68867e0eb627-cert\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.310484 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.310449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6963abc-7177-4904-ae7b-68867e0eb627-metrics-cert\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.310631 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.310494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c6963abc-7177-4904-ae7b-68867e0eb627-manager-config\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.310631 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.310532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6963abc-7177-4904-ae7b-68867e0eb627-cert\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.310631 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.310609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29pt\" (UniqueName: \"kubernetes.io/projected/c6963abc-7177-4904-ae7b-68867e0eb627-kube-api-access-m29pt\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.311329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.311310    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c6963abc-7177-4904-ae7b-68867e0eb627-manager-config\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.313060 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.313041    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6963abc-7177-4904-ae7b-68867e0eb627-metrics-cert\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.313119 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.313099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6963abc-7177-4904-ae7b-68867e0eb627-cert\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.334771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.334747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29pt\" (UniqueName: \"kubernetes.io/projected/c6963abc-7177-4904-ae7b-68867e0eb627-kube-api-access-m29pt\") pod \"lws-controller-manager-76d5cc9db8-jd7bq\" (UID: \"c6963abc-7177-4904-ae7b-68867e0eb627\") " pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.436848 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.436787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:27.568654 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:27.568630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"]
Jul 07 13:21:27.571280 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:21:27.571257    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6963abc_7177_4904_ae7b_68867e0eb627.slice/crio-d1c54a231e9b0c5ebd1d592586890cbeaac085def9ca4c7d4e648ce3ab3bdda7 WatchSource:0}: Error finding container d1c54a231e9b0c5ebd1d592586890cbeaac085def9ca4c7d4e648ce3ab3bdda7: Status 404 returned error can't find the container with id d1c54a231e9b0c5ebd1d592586890cbeaac085def9ca4c7d4e648ce3ab3bdda7
Jul 07 13:21:28.178870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:28.178833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq" event={"ID":"c6963abc-7177-4904-ae7b-68867e0eb627","Type":"ContainerStarted","Data":"d1c54a231e9b0c5ebd1d592586890cbeaac085def9ca4c7d4e648ce3ab3bdda7"}
Jul 07 13:21:28.660246 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:28.660209    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:30.188585 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:30.188491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq" event={"ID":"c6963abc-7177-4904-ae7b-68867e0eb627","Type":"ContainerStarted","Data":"65b8bca0047b315ccbe164f6a0ca488869a2b42a2d3d152b729921b8455c7416"}
Jul 07 13:21:30.188962 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:30.188598    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:30.208628 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:30.208584    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq" podStartSLOduration=0.980378574 podStartE2EDuration="3.208571146s" totalImagesPullingTime="2.228192572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:21:30.206620943 +0000 UTC m=+296.086992254" watchObservedRunningTime="2026-07-07 13:21:30.208571146 +0000 UTC m=+296.088942443"
Jul 07 13:21:32.609232 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:32.609193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:32.703070 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:32.703033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:33.332279 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.332243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:33.550103 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.550068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:33.660074 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.659999    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:33.705328 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.702937    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"]
Jul 07 13:21:33.706240 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.706219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.708839 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.708815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 13:21:33.708969 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.708816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 13:21:33.709088 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.709070    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 13:21:33.709194 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.709121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vc28\""
Jul 07 13:21:33.709268 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.709237    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 13:21:33.723608 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.723583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"]
Jul 07 13:21:33.762658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.762635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0522fb53-79eb-47f6-ac7a-2553b6d7c868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.762769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.762671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0522fb53-79eb-47f6-ac7a-2553b6d7c868-manifests\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.762769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.762712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mbj\" (UniqueName: \"kubernetes.io/projected/0522fb53-79eb-47f6-ac7a-2553b6d7c868-kube-api-access-j8mbj\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.762851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.762775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0522fb53-79eb-47f6-ac7a-2553b6d7c868-webhook-cert\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.762851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.762815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0522fb53-79eb-47f6-ac7a-2553b6d7c868-tmp\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.817516 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.817493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:33.864051 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.864028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0522fb53-79eb-47f6-ac7a-2553b6d7c868-tmp\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.864145 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.864063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0522fb53-79eb-47f6-ac7a-2553b6d7c868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.864145 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.864092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0522fb53-79eb-47f6-ac7a-2553b6d7c868-manifests\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.864145 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.864133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mbj\" (UniqueName: \"kubernetes.io/projected/0522fb53-79eb-47f6-ac7a-2553b6d7c868-kube-api-access-j8mbj\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.864315 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.864197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0522fb53-79eb-47f6-ac7a-2553b6d7c868-webhook-cert\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.864520 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.864503    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0522fb53-79eb-47f6-ac7a-2553b6d7c868-manifests\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.866428 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.866410    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0522fb53-79eb-47f6-ac7a-2553b6d7c868-tmp\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.866636 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.866617    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0522fb53-79eb-47f6-ac7a-2553b6d7c868-webhook-cert\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.866805 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.866784    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0522fb53-79eb-47f6-ac7a-2553b6d7c868-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.875142 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.875124    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mbj\" (UniqueName: \"kubernetes.io/projected/0522fb53-79eb-47f6-ac7a-2553b6d7c868-kube-api-access-j8mbj\") pod \"opendatahub-operator-controller-manager-69fbc88fbc-q5ghb\" (UID: \"0522fb53-79eb-47f6-ac7a-2553b6d7c868\") " pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:33.918142 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:33.918075    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:34.017290 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.017260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:34.026924 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.026894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:34.129852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.129815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:34.164616 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.164541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"]
Jul 07 13:21:34.166987 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:21:34.166963    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0522fb53_79eb_47f6_ac7a_2553b6d7c868.slice/crio-5cbc3c0a7f565aaca3fde587422f6c2a2304e0d2e1bac86e49263a5092eb4875 WatchSource:0}: Error finding container 5cbc3c0a7f565aaca3fde587422f6c2a2304e0d2e1bac86e49263a5092eb4875: Status 404 returned error can't find the container with id 5cbc3c0a7f565aaca3fde587422f6c2a2304e0d2e1bac86e49263a5092eb4875
Jul 07 13:21:34.207438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.207413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb" event={"ID":"0522fb53-79eb-47f6-ac7a-2553b6d7c868","Type":"ContainerStarted","Data":"5cbc3c0a7f565aaca3fde587422f6c2a2304e0d2e1bac86e49263a5092eb4875"}
Jul 07 13:21:34.226679 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.226650    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:34.327436 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.327403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:34.584237 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:34.584212    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 13:21:38.228410 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:38.228373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb" event={"ID":"0522fb53-79eb-47f6-ac7a-2553b6d7c868","Type":"ContainerStarted","Data":"db629bd30bfcdb458d807d862ab36eb6ff17b6d864dac8353520766b12349f63"}
Jul 07 13:21:39.234284 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:39.234252    2581 generic.go:356] "Generic (PLEG): container finished" podID="0522fb53-79eb-47f6-ac7a-2553b6d7c868" containerID="db629bd30bfcdb458d807d862ab36eb6ff17b6d864dac8353520766b12349f63" exitCode=0
Jul 07 13:21:39.234650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:39.234288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb" event={"ID":"0522fb53-79eb-47f6-ac7a-2553b6d7c868","Type":"ContainerDied","Data":"db629bd30bfcdb458d807d862ab36eb6ff17b6d864dac8353520766b12349f63"}
Jul 07 13:21:39.234826 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:39.234809    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 13:21:40.240725 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:40.240688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb" event={"ID":"0522fb53-79eb-47f6-ac7a-2553b6d7c868","Type":"ContainerStarted","Data":"8e2652f388063097330ad179756b7135e594270735e6d9bb6cfda1c3aab55fe0"}
Jul 07 13:21:40.241072 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:40.240793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:40.263521 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:40.263474    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb" podStartSLOduration=2.881598372 podStartE2EDuration="7.263461041s" totalImagesPullingTime="3.39009736s" totalInitContainerRuntime="991.765309ms" isStatefulPod=false podCreationTimestamp="2026-07-07 13:21:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:21:40.261978501 +0000 UTC m=+306.142349795" watchObservedRunningTime="2026-07-07 13:21:40.263461041 +0000 UTC m=+306.143832335"
Jul 07 13:21:41.195484 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:41.195454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76d5cc9db8-jd7bq"
Jul 07 13:21:45.716066 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:45.716033    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:45.818905 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:45.818876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:51.249931 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:51.249899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69fbc88fbc-q5ghb"
Jul 07 13:21:52.905016 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:52.904961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:53.014186 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.014127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:53.403317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.403283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:53.610832 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.610781    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"]
Jul 07 13:21:53.625248 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.625223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.628411 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.628390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mff25\""
Jul 07 13:21:53.629547 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.629530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 13:21:53.629628 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.629531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 13:21:53.629889 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.629870    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 13:21:53.630010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.629950    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 13:21:53.637696 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.637674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"]
Jul 07 13:21:53.689204 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.689123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:53.732526 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.732498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540609e-e8e9-483f-8ce9-57ff0fc14da7-tls-certs\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.732650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.732551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5540609e-e8e9-483f-8ce9-57ff0fc14da7-tmp\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.732650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.732575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwk6\" (UniqueName: \"kubernetes.io/projected/5540609e-e8e9-483f-8ce9-57ff0fc14da7-kube-api-access-9kwk6\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.798929 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.798894    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:53.833379 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.833350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540609e-e8e9-483f-8ce9-57ff0fc14da7-tls-certs\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.833540 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.833399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5540609e-e8e9-483f-8ce9-57ff0fc14da7-tmp\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.833540 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.833427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwk6\" (UniqueName: \"kubernetes.io/projected/5540609e-e8e9-483f-8ce9-57ff0fc14da7-kube-api-access-9kwk6\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.835758 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.835731    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5540609e-e8e9-483f-8ce9-57ff0fc14da7-tmp\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.835987 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.835968    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5540609e-e8e9-483f-8ce9-57ff0fc14da7-tls-certs\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.844839 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.844818    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwk6\" (UniqueName: \"kubernetes.io/projected/5540609e-e8e9-483f-8ce9-57ff0fc14da7-kube-api-access-9kwk6\") pod \"kube-auth-proxy-577c5f8c7-qxgnv\" (UID: \"5540609e-e8e9-483f-8ce9-57ff0fc14da7\") " pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:53.872682 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.872661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:53.937399 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:53.937376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"
Jul 07 13:21:54.085785 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:54.085762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv"]
Jul 07 13:21:54.087959 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:21:54.087927    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5540609e_e8e9_483f_8ce9_57ff0fc14da7.slice/crio-9c6190bdea5e273c7ea6729f60871dd7774e7c53cdac3ffb7d6bf5d1b07c4877 WatchSource:0}: Error finding container 9c6190bdea5e273c7ea6729f60871dd7774e7c53cdac3ffb7d6bf5d1b07c4877: Status 404 returned error can't find the container with id 9c6190bdea5e273c7ea6729f60871dd7774e7c53cdac3ffb7d6bf5d1b07c4877
Jul 07 13:21:54.099724 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:54.099666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:54.306196 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:54.304651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv" event={"ID":"5540609e-e8e9-483f-8ce9-57ff0fc14da7","Type":"ContainerStarted","Data":"9c6190bdea5e273c7ea6729f60871dd7774e7c53cdac3ffb7d6bf5d1b07c4877"}
Jul 07 13:21:54.532802 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:54.532772    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:54.778025 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:54.777960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:54.877255 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:54.877224    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:55.795393 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:55.795358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:58.315181 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:58.315136    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:21:58.324143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:58.324111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv" event={"ID":"5540609e-e8e9-483f-8ce9-57ff0fc14da7","Type":"ContainerStarted","Data":"11ef61fabad42fb87e791ffced8f1b527b59dfbb8e8d51974490e04435487d42"}
Jul 07 13:21:58.343784 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:58.343739    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-577c5f8c7-qxgnv" podStartSLOduration=2.130265806 podStartE2EDuration="5.343724483s" totalImagesPullingTime="3.213458677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:21:58.341635985 +0000 UTC m=+324.222007279" watchObservedRunningTime="2026-07-07 13:21:58.343724483 +0000 UTC m=+324.224095777"
Jul 07 13:21:58.414074 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:21:58.414042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:00.280253 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:00.280218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:00.456268 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:00.456239    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:01.189928 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:01.189896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:01.305432 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:01.305405    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:01.703581 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:01.703550    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:03.992562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:03.992533    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:04.119061 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:04.119022    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:04.208697 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:04.208664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:04.314275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:04.314175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:04.442443 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:04.442415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:54.438035 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:54.438001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:54.579870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:54.579830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:22:54.689068 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:22:54.688991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:17.512893 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:17.512862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:17.612549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:17.612515    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:19.450871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:19.450831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:19.503637 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:19.503606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:22.216376 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:22.216341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:25.826401 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:25.826357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:28.841582 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:28.841547    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:30.329737 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:30.329703    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:30.624113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:30.624028    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:31.204222 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:31.204189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:33.680374 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:33.680342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:33.779133 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:33.779103    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:34.642898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:34.642862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:36.481276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:36.481244    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:38.023093 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:38.023045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:38.453632 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:38.453544    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:40.091473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.091441    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"]
Jul 07 13:23:40.094729 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.094709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.097743 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.097720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 07 13:23:40.097852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.097736    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 13:23:40.098488 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.098471    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 07 13:23:40.098874 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.098857    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ms98c\""
Jul 07 13:23:40.098969 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.098862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 13:23:40.104491 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.104471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"]
Jul 07 13:23:40.112362 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.112343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:40.193573 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.193552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csh8\" (UniqueName: \"kubernetes.io/projected/13d458d1-fdf4-4a80-9666-0e5b949c93f3-kube-api-access-5csh8\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.193677 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.193613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.193677 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.193665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13d458d1-fdf4-4a80-9666-0e5b949c93f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.294450 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.294426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csh8\" (UniqueName: \"kubernetes.io/projected/13d458d1-fdf4-4a80-9666-0e5b949c93f3-kube-api-access-5csh8\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.294553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.294513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.294553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.294548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13d458d1-fdf4-4a80-9666-0e5b949c93f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.294649 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:40.294632    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 07 13:23:40.294699 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:40.294689    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert podName:13d458d1-fdf4-4a80-9666-0e5b949c93f3 nodeName:}" failed. No retries permitted until 2026-07-07 13:23:40.794673427 +0000 UTC m=+426.675044699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-tsdhk" (UID: "13d458d1-fdf4-4a80-9666-0e5b949c93f3") : secret "plugin-serving-cert" not found
Jul 07 13:23:40.295140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.295123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/13d458d1-fdf4-4a80-9666-0e5b949c93f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.302495 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.302467    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csh8\" (UniqueName: \"kubernetes.io/projected/13d458d1-fdf4-4a80-9666-0e5b949c93f3-kube-api-access-5csh8\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.798240 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:40.798213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:40.798439 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:40.798357    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 07 13:23:40.798439 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:40.798423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert podName:13d458d1-fdf4-4a80-9666-0e5b949c93f3 nodeName:}" failed. No retries permitted until 2026-07-07 13:23:41.798409696 +0000 UTC m=+427.678780971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-tsdhk" (UID: "13d458d1-fdf4-4a80-9666-0e5b949c93f3") : secret "plugin-serving-cert" not found
Jul 07 13:23:41.002027 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:41.002000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:41.612095 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:41.612047    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:41.713008 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:41.712964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:41.806778 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:41.806753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:41.813119 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:41.810427    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/13d458d1-fdf4-4a80-9666-0e5b949c93f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-tsdhk\" (UID: \"13d458d1-fdf4-4a80-9666-0e5b949c93f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:41.904391 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:41.904327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"
Jul 07 13:23:42.035665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:42.035645    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk"]
Jul 07 13:23:42.038281 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:23:42.038252    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13d458d1_fdf4_4a80_9666_0e5b949c93f3.slice/crio-c9c3904d34ba3a827e27756f709390127c244db4e87f266f288b73ad8641700e WatchSource:0}: Error finding container c9c3904d34ba3a827e27756f709390127c244db4e87f266f288b73ad8641700e: Status 404 returned error can't find the container with id c9c3904d34ba3a827e27756f709390127c244db4e87f266f288b73ad8641700e
Jul 07 13:23:42.774620 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:42.774585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk" event={"ID":"13d458d1-fdf4-4a80-9666-0e5b949c93f3","Type":"ContainerStarted","Data":"c9c3904d34ba3a827e27756f709390127c244db4e87f266f288b73ad8641700e"}
Jul 07 13:23:49.909784 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:49.909746    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"]
Jul 07 13:23:49.913453 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:49.913431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:49.916188 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:49.916146    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2j7s9\""
Jul 07 13:23:49.927670 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:49.927647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"]
Jul 07 13:23:49.986443 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:49.986408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87gj\" (UniqueName: \"kubernetes.io/projected/0ffed87b-c46e-4559-b151-bb66e7cb637a-kube-api-access-z87gj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:49.986648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:49.986452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ffed87b-c46e-4559-b151-bb66e7cb637a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.007527 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.007499    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"]
Jul 07 13:23:50.007746 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:50.007727    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-z87gj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw" podUID="0ffed87b-c46e-4559-b151-bb66e7cb637a"
Jul 07 13:23:50.087206 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.087150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87gj\" (UniqueName: \"kubernetes.io/projected/0ffed87b-c46e-4559-b151-bb66e7cb637a-kube-api-access-z87gj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.087349 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.087219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ffed87b-c46e-4559-b151-bb66e7cb637a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.087625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.087594    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ffed87b-c46e-4559-b151-bb66e7cb637a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.095986 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.095958    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87gj\" (UniqueName: \"kubernetes.io/projected/0ffed87b-c46e-4559-b151-bb66e7cb637a-kube-api-access-z87gj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.529143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.529106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"]
Jul 07 13:23:50.535198 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.535150    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"]
Jul 07 13:23:50.602933 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.602900    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"]
Jul 07 13:23:50.605896 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.605872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:23:50.608570 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.608548    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6grr6\""
Jul 07 13:23:50.620892 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.620867    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"]
Jul 07 13:23:50.692653 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.692618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jdv\" (UniqueName: \"kubernetes.io/projected/1a1140db-844d-4c86-ba2b-47264b20f075-kube-api-access-68jdv\") pod \"limitador-operator-controller-manager-859d7b78bd-dp5l6\" (UID: \"1a1140db-844d-4c86-ba2b-47264b20f075\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:23:50.793587 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.793408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jdv\" (UniqueName: \"kubernetes.io/projected/1a1140db-844d-4c86-ba2b-47264b20f075-kube-api-access-68jdv\") pod \"limitador-operator-controller-manager-859d7b78bd-dp5l6\" (UID: \"1a1140db-844d-4c86-ba2b-47264b20f075\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:23:50.801961 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.801926    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jdv\" (UniqueName: \"kubernetes.io/projected/1a1140db-844d-4c86-ba2b-47264b20f075-kube-api-access-68jdv\") pod \"limitador-operator-controller-manager-859d7b78bd-dp5l6\" (UID: \"1a1140db-844d-4c86-ba2b-47264b20f075\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:23:50.815776 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.815751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.826610 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.826589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.829173 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:50.829129    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" is forbidden: User \"system:node:ip-10-0-133-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-157.ec2.internal' and this object" podUID="0ffed87b-c46e-4559-b151-bb66e7cb637a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:50.840828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.840778    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:50.894433 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.894394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z87gj\" (UniqueName: \"kubernetes.io/projected/0ffed87b-c46e-4559-b151-bb66e7cb637a-kube-api-access-z87gj\") pod \"0ffed87b-c46e-4559-b151-bb66e7cb637a\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") "
Jul 07 13:23:50.894583 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.894542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ffed87b-c46e-4559-b151-bb66e7cb637a-extensions-socket-volume\") pod \"0ffed87b-c46e-4559-b151-bb66e7cb637a\" (UID: \"0ffed87b-c46e-4559-b151-bb66e7cb637a\") "
Jul 07 13:23:50.894824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.894789    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ffed87b-c46e-4559-b151-bb66e7cb637a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0ffed87b-c46e-4559-b151-bb66e7cb637a" (UID: "0ffed87b-c46e-4559-b151-bb66e7cb637a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:23:50.897229 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.897204    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ffed87b-c46e-4559-b151-bb66e7cb637a-kube-api-access-z87gj" (OuterVolumeSpecName: "kube-api-access-z87gj") pod "0ffed87b-c46e-4559-b151-bb66e7cb637a" (UID: "0ffed87b-c46e-4559-b151-bb66e7cb637a"). InnerVolumeSpecName "kube-api-access-z87gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:23:50.920290 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.920199    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:23:50.940439 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.940403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:50.996306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.996272    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ffed87b-c46e-4559-b151-bb66e7cb637a-extensions-socket-volume\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:23:50.996306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:50.996309    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87gj\" (UniqueName: \"kubernetes.io/projected/0ffed87b-c46e-4559-b151-bb66e7cb637a-kube-api-access-z87gj\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:23:51.821328 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:51.821298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:51.823715 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:51.823673    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" is forbidden: User \"system:node:ip-10-0-133-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-157.ec2.internal' and this object" podUID="0ffed87b-c46e-4559-b151-bb66e7cb637a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:51.830792 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:23:51.830761    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-94tsw\" is forbidden: User \"system:node:ip-10-0-133-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-157.ec2.internal' and this object" podUID="0ffed87b-c46e-4559-b151-bb66e7cb637a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-94tsw"
Jul 07 13:23:52.762143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:52.762100    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ffed87b-c46e-4559-b151-bb66e7cb637a" path="/var/lib/kubelet/pods/0ffed87b-c46e-4559-b151-bb66e7cb637a/volumes"
Jul 07 13:23:53.836918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:53.836879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:53.938231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:53.938187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:56.442734 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:56.442651    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:23:56.540771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:23:56.540725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:00.297919 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:00.297876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:00.398367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:00.398333    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:05.070049 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:05.069953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"]
Jul 07 13:24:05.073070 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:24:05.073041    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1140db_844d_4c86_ba2b_47264b20f075.slice/crio-345604ed46bde985d15484899596e238e2abf82153df292627285db107f27181 WatchSource:0}: Error finding container 345604ed46bde985d15484899596e238e2abf82153df292627285db107f27181: Status 404 returned error can't find the container with id 345604ed46bde985d15484899596e238e2abf82153df292627285db107f27181
Jul 07 13:24:05.899514 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:05.899475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk" event={"ID":"13d458d1-fdf4-4a80-9666-0e5b949c93f3","Type":"ContainerStarted","Data":"1a9606c5d80d972530c7d39770418931145a9773447c1e63d1c952f811e674b6"}
Jul 07 13:24:05.900760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:05.900725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6" event={"ID":"1a1140db-844d-4c86-ba2b-47264b20f075","Type":"ContainerStarted","Data":"345604ed46bde985d15484899596e238e2abf82153df292627285db107f27181"}
Jul 07 13:24:05.928017 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:05.927974    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-tsdhk" podStartSLOduration=2.626903542 podStartE2EDuration="25.927959848s" totalImagesPullingTime="23.301056306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:24:05.926602301 +0000 UTC m=+451.806973589" watchObservedRunningTime="2026-07-07 13:24:05.927959848 +0000 UTC m=+451.808331146"
Jul 07 13:24:09.919605 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:09.919570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6" event={"ID":"1a1140db-844d-4c86-ba2b-47264b20f075","Type":"ContainerStarted","Data":"9ced4fb8f5153d85d84ec783a44412a3087c9bcf2dbea28dc1117e061fd9caa9"}
Jul 07 13:24:09.919976 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:09.919645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:24:09.938774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:09.938722    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6" podStartSLOduration=15.954451811 podStartE2EDuration="19.938706875s" totalImagesPullingTime="3.984255064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:24:09.937845738 +0000 UTC m=+455.818217042" watchObservedRunningTime="2026-07-07 13:24:09.938706875 +0000 UTC m=+455.819078168"
Jul 07 13:24:11.834758 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:11.834724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:11.933625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:11.933594    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:12.176617 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:12.176546    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:13.111535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:13.111458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:13.176329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:13.176300    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:17.035458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:17.035424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:17.135860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:17.135827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:19.288328 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:19.288289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:19.388435 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:19.388401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:20.927138 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:20.927110    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dp5l6"
Jul 07 13:24:30.290771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:30.290737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:30.343954 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:30.343925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:31.273229 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:31.273196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:31.333026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:31.333002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:32.720113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:32.720081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:32.836532 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:32.836501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:33.337298 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:33.337268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:33.527936 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:33.527904    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:33.589354 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:33.589294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:42.216616 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:42.216581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:42.316926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:42.316897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:42.372636 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:42.372604    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:24:50.831177 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:24:50.831138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:03.552254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:03.552220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:04.067963 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:04.067928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:10.760534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:10.760494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:10.853410 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:10.853372    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:11.335714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:11.335682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:14.334618 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:14.334576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:14.429656 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:14.429612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:44.432618 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:44.432531    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:44.849969 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:44.849906    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:45.257524 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.257496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:45.357801 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.357773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:45.358214 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.358194    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-4nrs6"]
Jul 07 13:25:45.360517 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.360500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.364384 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.364367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 13:25:45.364485 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.364392    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vc4lg\""
Jul 07 13:25:45.372273 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.372254    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-4nrs6"]
Jul 07 13:25:45.397491 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.397459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288a9386-9eb8-4146-9d36-1f1da3098d34-plugins-config-volume\") pod \"payload-pre-processing-7964457696-4nrs6\" (UID: \"288a9386-9eb8-4146-9d36-1f1da3098d34\") " pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.397586 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.397575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxznn\" (UniqueName: \"kubernetes.io/projected/288a9386-9eb8-4146-9d36-1f1da3098d34-kube-api-access-qxznn\") pod \"payload-pre-processing-7964457696-4nrs6\" (UID: \"288a9386-9eb8-4146-9d36-1f1da3098d34\") " pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.498700 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.498666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxznn\" (UniqueName: \"kubernetes.io/projected/288a9386-9eb8-4146-9d36-1f1da3098d34-kube-api-access-qxznn\") pod \"payload-pre-processing-7964457696-4nrs6\" (UID: \"288a9386-9eb8-4146-9d36-1f1da3098d34\") " pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.499057 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.498730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288a9386-9eb8-4146-9d36-1f1da3098d34-plugins-config-volume\") pod \"payload-pre-processing-7964457696-4nrs6\" (UID: \"288a9386-9eb8-4146-9d36-1f1da3098d34\") " pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.499386 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.499368    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/288a9386-9eb8-4146-9d36-1f1da3098d34-plugins-config-volume\") pod \"payload-pre-processing-7964457696-4nrs6\" (UID: \"288a9386-9eb8-4146-9d36-1f1da3098d34\") " pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.506531 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.506510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxznn\" (UniqueName: \"kubernetes.io/projected/288a9386-9eb8-4146-9d36-1f1da3098d34-kube-api-access-qxznn\") pod \"payload-pre-processing-7964457696-4nrs6\" (UID: \"288a9386-9eb8-4146-9d36-1f1da3098d34\") " pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.535401 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.535341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:45.670470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.670441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:45.800288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:45.800199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-4nrs6"]
Jul 07 13:25:45.803212 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:25:45.803185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a9386_9eb8_4146_9d36_1f1da3098d34.slice/crio-ee24a58ac07ca4d7b2b6becd1ae451bfab19f65829df5364f389d7e65952bf17 WatchSource:0}: Error finding container ee24a58ac07ca4d7b2b6becd1ae451bfab19f65829df5364f389d7e65952bf17: Status 404 returned error can't find the container with id ee24a58ac07ca4d7b2b6becd1ae451bfab19f65829df5364f389d7e65952bf17
Jul 07 13:25:46.334630 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:46.334592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6" event={"ID":"288a9386-9eb8-4146-9d36-1f1da3098d34","Type":"ContainerStarted","Data":"ee24a58ac07ca4d7b2b6becd1ae451bfab19f65829df5364f389d7e65952bf17"}
Jul 07 13:25:47.341352 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:47.341309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6" event={"ID":"288a9386-9eb8-4146-9d36-1f1da3098d34","Type":"ContainerStarted","Data":"31a820078268f854466794365f457a71d3077e6a6cea63d7d2710f5e319ea7b1"}
Jul 07 13:25:47.341802 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:47.341401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:25:47.357744 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:47.357696    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6" podStartSLOduration=1.031759547 podStartE2EDuration="2.357682945s" totalImagesPullingTime="1.325923398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:25:47.35565489 +0000 UTC m=+553.236026210" watchObservedRunningTime="2026-07-07 13:25:47.357682945 +0000 UTC m=+553.238054259"
Jul 07 13:25:55.544656 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:55.544626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:55.637713 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:55.637683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:25:58.347645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:25:58.347609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-4nrs6"
Jul 07 13:26:09.406843 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:09.406805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:26:13.253967 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:13.253937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:26:27.293935 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.293899    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"]
Jul 07 13:26:27.296447 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.296425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.299131 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.299103    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 13:26:27.299395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.299381    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 13:26:27.300708 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.300687    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qgt7b\""
Jul 07 13:26:27.300817 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.300750    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 07 13:26:27.306594 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.306571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"]
Jul 07 13:26:27.324582 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.324559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:26:27.433366 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.433341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.433470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.433383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j44rm\" (UniqueName: \"kubernetes.io/projected/6623c51c-1307-4425-9966-4a6f11ef005b-kube-api-access-j44rm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.433470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.433407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.433556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.433467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.433556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.433491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.433556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.433544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6623c51c-1307-4425-9966-4a6f11ef005b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534053 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534053 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534253 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6623c51c-1307-4425-9966-4a6f11ef005b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534253 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534364 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j44rm\" (UniqueName: \"kubernetes.io/projected/6623c51c-1307-4425-9966-4a6f11ef005b-kube-api-access-j44rm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534364 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534513 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534493    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534529    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.534632 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.534593    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.536521 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.536505    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6623c51c-1307-4425-9966-4a6f11ef005b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.536769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.536753    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6623c51c-1307-4425-9966-4a6f11ef005b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.542773 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.542751    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j44rm\" (UniqueName: \"kubernetes.io/projected/6623c51c-1307-4425-9966-4a6f11ef005b-kube-api-access-j44rm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9xdkv\" (UID: \"6623c51c-1307-4425-9966-4a6f11ef005b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.607892 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.607797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:27.742497 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:27.742420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"]
Jul 07 13:26:27.744870 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:26:27.744841    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6623c51c_1307_4425_9966_4a6f11ef005b.slice/crio-edb4e057933a31a0f3e53a6ac2ebaa0b7de214df51db0c522dc5cbf1ba090c92 WatchSource:0}: Error finding container edb4e057933a31a0f3e53a6ac2ebaa0b7de214df51db0c522dc5cbf1ba090c92: Status 404 returned error can't find the container with id edb4e057933a31a0f3e53a6ac2ebaa0b7de214df51db0c522dc5cbf1ba090c92
Jul 07 13:26:28.526321 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:28.526278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv" event={"ID":"6623c51c-1307-4425-9966-4a6f11ef005b","Type":"ContainerStarted","Data":"edb4e057933a31a0f3e53a6ac2ebaa0b7de214df51db0c522dc5cbf1ba090c92"}
Jul 07 13:26:29.531916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:29.531882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv" event={"ID":"6623c51c-1307-4425-9966-4a6f11ef005b","Type":"ContainerStarted","Data":"821ef0bd24304881a41fd476b97f63787db19266580507f9a0a010780d3dfd49"}
Jul 07 13:26:29.550973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:29.550922    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv" podStartSLOduration=1.541238273 podStartE2EDuration="2.550906509s" totalImagesPullingTime="1.009668236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:26:29.549902819 +0000 UTC m=+595.430274113" watchObservedRunningTime="2026-07-07 13:26:29.550906509 +0000 UTC m=+595.431277804"
Jul 07 13:26:37.608701 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:37.608666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:37.611392 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:37.608734    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:37.621079 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:37.621057    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:38.583473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:38.583438    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9xdkv"
Jul 07 13:26:53.337781 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:26:53.337746    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:01.793045 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:01.792959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:14.420085 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:14.420048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:28.100390 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.100355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"]
Jul 07 13:27:28.103339 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.103323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.106138 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.106119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 13:27:28.112833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.112809    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"]
Jul 07 13:27:28.132350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.132325    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:28.244323 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.244295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.244442 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.244335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.244442 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.244368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.244442 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.244423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.244607 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.244497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.244607 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.244525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv55\" (UniqueName: \"kubernetes.io/projected/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-kube-api-access-jxv55\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.344986 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.344958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv55\" (UniqueName: \"kubernetes.io/projected/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-kube-api-access-jxv55\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345101 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345101 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345101 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345314 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345382 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345472 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345548 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345531    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.345586 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.345572    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.347489 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.347466    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.347714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.347695    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.352995 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.352942    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv55\" (UniqueName: \"kubernetes.io/projected/a98a46bf-1d6d-47c2-9259-fed852d7c1fe-kube-api-access-jxv55\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c\" (UID: \"a98a46bf-1d6d-47c2-9259-fed852d7c1fe\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.413603 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.413580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:28.551498 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.551476    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"]
Jul 07 13:27:28.554041 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:27:28.554014    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda98a46bf_1d6d_47c2_9259_fed852d7c1fe.slice/crio-727782c35f20088dbd86e6a11ffe98d2e042579df73a7f00a32800122b7e8215 WatchSource:0}: Error finding container 727782c35f20088dbd86e6a11ffe98d2e042579df73a7f00a32800122b7e8215: Status 404 returned error can't find the container with id 727782c35f20088dbd86e6a11ffe98d2e042579df73a7f00a32800122b7e8215
Jul 07 13:27:28.555414 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.555396    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 13:27:28.796899 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:28.796870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c" event={"ID":"a98a46bf-1d6d-47c2-9259-fed852d7c1fe","Type":"ContainerStarted","Data":"727782c35f20088dbd86e6a11ffe98d2e042579df73a7f00a32800122b7e8215"}
Jul 07 13:27:29.803229 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:29.803197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c" event={"ID":"a98a46bf-1d6d-47c2-9259-fed852d7c1fe","Type":"ContainerStarted","Data":"2502ff5a48caa2d129dbbae02b0ed019c31801c8fd5cb84aa458fb7a64e2cf74"}
Jul 07 13:27:29.822276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:29.822231    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c" podStartSLOduration=1.551443666 podStartE2EDuration="1.822214754s" totalImagesPullingTime="270.771088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:27:29.821406679 +0000 UTC m=+655.701777975" watchObservedRunningTime="2026-07-07 13:27:29.822214754 +0000 UTC m=+655.702586048"
Jul 07 13:27:38.373063 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:38.373031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:38.414645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:38.414615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:38.414770 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:38.414672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:38.427059 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:38.427037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:38.430846 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:38.430819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:38.851909 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:38.851875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c"
Jul 07 13:27:42.134423 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:42.134395    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:42.194071 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:42.194038    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:27:56.508094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:27:56.508054    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:28:03.848267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:28:03.848238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:28:03.898777 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:28:03.898749    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:28:11.086118 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:28:11.086086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:29:11.389750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:29:11.389718    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:29:11.496569 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:29:11.496536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:29:12.586122 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:29:12.586077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:29:12.679819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:29:12.679792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:30:22.725449 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:30:22.725413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:30:23.959482 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:30:23.959449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:30:47.609835 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:30:47.609794    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:30:47.710840 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:30:47.710806    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:30:57.833541 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:30:57.833502    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:30:57.890107 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:30:57.890081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:03.676294 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:03.676237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:03.774898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:03.774866    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:03.933922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:03.933843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:09.680785 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:09.680750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:09.741116 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:09.741088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:09.826852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:09.826816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:09.923536 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:09.923508    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:10.060305 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:10.060276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:15.673200 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:15.673156    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:31:15.755562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:31:15.755541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:34:11.955136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:34:11.955102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:36:01.070733 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:36:01.070644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:36:24.551298 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:36:24.551264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:36:24.649881 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:36:24.649847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:37:25.551612 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:37:25.551528    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:41:36.144766 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:41:36.144728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:41:53.604237 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:41:53.604189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:41:58.987746 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:41:58.987669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:07.309835 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:07.309790    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:12.610731 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:12.610699    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:15.582483 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:15.582428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:15.699036 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:15.698994    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:16.075937 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:16.075899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:21.832397 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:21.832359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:21.934773 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:21.934739    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:22.061502 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:22.061474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:22.159535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:22.159458    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:23.231717 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.231686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:23.314926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.314889    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"]
Jul 07 13:42:23.318377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.318355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.321283 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.321264    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:42:23.321508 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.321490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-0fbb7bc5-serving-cert\""
Jul 07 13:42:23.330211 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.330151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"]
Jul 07 13:42:23.337508 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.337486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:23.406403 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.406382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b23367-e6e2-48ca-9939-90a9b218fa97-maas-api-tls\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.406540 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.406430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rd7\" (UniqueName: \"kubernetes.io/projected/a0b23367-e6e2-48ca-9939-90a9b218fa97-kube-api-access-d9rd7\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.406540 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.406465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b23367-e6e2-48ca-9939-90a9b218fa97-openshift-service-ca\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.506887 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.506857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rd7\" (UniqueName: \"kubernetes.io/projected/a0b23367-e6e2-48ca-9939-90a9b218fa97-kube-api-access-d9rd7\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.507017 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.506900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b23367-e6e2-48ca-9939-90a9b218fa97-openshift-service-ca\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.507017 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.507002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b23367-e6e2-48ca-9939-90a9b218fa97-maas-api-tls\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.507545 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.507524    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b23367-e6e2-48ca-9939-90a9b218fa97-openshift-service-ca\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.509558 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.509541    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b23367-e6e2-48ca-9939-90a9b218fa97-maas-api-tls\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.515582 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.515562    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rd7\" (UniqueName: \"kubernetes.io/projected/a0b23367-e6e2-48ca-9939-90a9b218fa97-kube-api-access-d9rd7\") pod \"maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") " pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.629315 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.629290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:23.708571 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.708542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:23.761429 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.761402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"]
Jul 07 13:42:23.764402 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:42:23.764378    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b23367_e6e2_48ca_9939_90a9b218fa97.slice/crio-64ad3180c572c0c77e222feb84a4b910cc87de90d25ae14ac85250e880f0936b WatchSource:0}: Error finding container 64ad3180c572c0c77e222feb84a4b910cc87de90d25ae14ac85250e880f0936b: Status 404 returned error can't find the container with id 64ad3180c572c0c77e222feb84a4b910cc87de90d25ae14ac85250e880f0936b
Jul 07 13:42:23.765819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.765798    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 13:42:23.791580 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.791551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv" event={"ID":"a0b23367-e6e2-48ca-9939-90a9b218fa97","Type":"ContainerStarted","Data":"64ad3180c572c0c77e222feb84a4b910cc87de90d25ae14ac85250e880f0936b"}
Jul 07 13:42:23.809922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.809897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:23.908457 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:23.908429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:26.055830 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:26.055798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:26.810635 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:26.810590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv" event={"ID":"a0b23367-e6e2-48ca-9939-90a9b218fa97","Type":"ContainerStarted","Data":"678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5"}
Jul 07 13:42:26.810789 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:26.810778    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:26.828151 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:26.828113    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv" podStartSLOduration=1.315564557 podStartE2EDuration="3.82810126s" totalImagesPullingTime="2.512536703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:42:26.827072391 +0000 UTC m=+1552.707443708" watchObservedRunningTime="2026-07-07 13:42:26.82810126 +0000 UTC m=+1552.708472554"
Jul 07 13:42:28.341604 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:28.341565    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:28.584436 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:28.584401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:28.633404 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:28.633328    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"]
Jul 07 13:42:28.684652 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:28.684617    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:28.741369 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:28.741331    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:28.819583 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:28.819542    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv" podUID="a0b23367-e6e2-48ca-9939-90a9b218fa97" containerName="maas-api" containerID="cri-o://678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5" gracePeriod=30
Jul 07 13:42:29.065999 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.065975    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:29.155658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.155630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b23367-e6e2-48ca-9939-90a9b218fa97-maas-api-tls\") pod \"a0b23367-e6e2-48ca-9939-90a9b218fa97\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") "
Jul 07 13:42:29.155804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.155696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b23367-e6e2-48ca-9939-90a9b218fa97-openshift-service-ca\") pod \"a0b23367-e6e2-48ca-9939-90a9b218fa97\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") "
Jul 07 13:42:29.155804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.155744    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9rd7\" (UniqueName: \"kubernetes.io/projected/a0b23367-e6e2-48ca-9939-90a9b218fa97-kube-api-access-d9rd7\") pod \"a0b23367-e6e2-48ca-9939-90a9b218fa97\" (UID: \"a0b23367-e6e2-48ca-9939-90a9b218fa97\") "
Jul 07 13:42:29.156096 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.156071    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b23367-e6e2-48ca-9939-90a9b218fa97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0b23367-e6e2-48ca-9939-90a9b218fa97" (UID: "a0b23367-e6e2-48ca-9939-90a9b218fa97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:42:29.157906 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.157886    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b23367-e6e2-48ca-9939-90a9b218fa97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0b23367-e6e2-48ca-9939-90a9b218fa97" (UID: "a0b23367-e6e2-48ca-9939-90a9b218fa97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:42:29.157962 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.157930    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b23367-e6e2-48ca-9939-90a9b218fa97-kube-api-access-d9rd7" (OuterVolumeSpecName: "kube-api-access-d9rd7") pod "a0b23367-e6e2-48ca-9939-90a9b218fa97" (UID: "a0b23367-e6e2-48ca-9939-90a9b218fa97"). InnerVolumeSpecName "kube-api-access-d9rd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:42:29.256538 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.256515    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rd7\" (UniqueName: \"kubernetes.io/projected/a0b23367-e6e2-48ca-9939-90a9b218fa97-kube-api-access-d9rd7\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:29.256538 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.256537    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b23367-e6e2-48ca-9939-90a9b218fa97-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:29.256675 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.256546    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b23367-e6e2-48ca-9939-90a9b218fa97-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:29.824982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.824948    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0b23367-e6e2-48ca-9939-90a9b218fa97" containerID="678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5" exitCode=0
Jul 07 13:42:29.825464 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.824999    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"
Jul 07 13:42:29.825464 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.825017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv" event={"ID":"a0b23367-e6e2-48ca-9939-90a9b218fa97","Type":"ContainerDied","Data":"678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5"}
Jul 07 13:42:29.825464 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.825057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv" event={"ID":"a0b23367-e6e2-48ca-9939-90a9b218fa97","Type":"ContainerDied","Data":"64ad3180c572c0c77e222feb84a4b910cc87de90d25ae14ac85250e880f0936b"}
Jul 07 13:42:29.825464 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.825078    2581 scope.go:117] "RemoveContainer" containerID="678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5"
Jul 07 13:42:29.834255 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.834236    2581 scope.go:117] "RemoveContainer" containerID="678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5"
Jul 07 13:42:29.834511 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:42:29.834491    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5\": container with ID starting with 678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5 not found: ID does not exist" containerID="678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5"
Jul 07 13:42:29.834553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.834521    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5"} err="failed to get container status \"678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5\": rpc error: code = NotFound desc = could not find container \"678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5\": container with ID starting with 678e5e49cb13dce79887759274493865338aa7852eaab3228b43e4ec3e3884c5 not found: ID does not exist"
Jul 07 13:42:29.845990 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.845963    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"]
Jul 07 13:42:29.849760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:29.849738    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-0fbb7bc5-848948776-zqtjv"]
Jul 07 13:42:30.023377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:30.023350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:30.761183 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:30.761118    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0b23367-e6e2-48ca-9939-90a9b218fa97" path="/var/lib/kubelet/pods/a0b23367-e6e2-48ca-9939-90a9b218fa97/volumes"
Jul 07 13:42:34.352346 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:34.352312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:34.427892 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:34.427854    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:34.581781 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:34.581753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:34.653687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:34.653612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:35.671244 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.671203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:35.734666 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.734631    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"]
Jul 07 13:42:35.735007 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.734994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b23367-e6e2-48ca-9939-90a9b218fa97" containerName="maas-api"
Jul 07 13:42:35.735046 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.735009    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b23367-e6e2-48ca-9939-90a9b218fa97" containerName="maas-api"
Jul 07 13:42:35.735082 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.735076    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0b23367-e6e2-48ca-9939-90a9b218fa97" containerName="maas-api"
Jul 07 13:42:35.741327 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.741304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.744873 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.744848    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:42:35.744994 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.744856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-ae77d6a7-serving-cert\""
Jul 07 13:42:35.752370 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.752347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"]
Jul 07 13:42:35.761498 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.761473    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:35.815728 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.815701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtvq\" (UniqueName: \"kubernetes.io/projected/da9e1173-96a0-410e-a363-6bcab0841d36-kube-api-access-vhtvq\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.815834 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.815779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9e1173-96a0-410e-a363-6bcab0841d36-maas-api-tls\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.815880 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.815830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9e1173-96a0-410e-a363-6bcab0841d36-openshift-service-ca\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.836237 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.836189    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:35.916474 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.916446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9e1173-96a0-410e-a363-6bcab0841d36-maas-api-tls\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.916645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.916551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9e1173-96a0-410e-a363-6bcab0841d36-openshift-service-ca\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.916645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.916594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtvq\" (UniqueName: \"kubernetes.io/projected/da9e1173-96a0-410e-a363-6bcab0841d36-kube-api-access-vhtvq\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.917149 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.917132    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9e1173-96a0-410e-a363-6bcab0841d36-openshift-service-ca\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.918857 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.918835    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9e1173-96a0-410e-a363-6bcab0841d36-maas-api-tls\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.925787 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.925730    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtvq\" (UniqueName: \"kubernetes.io/projected/da9e1173-96a0-410e-a363-6bcab0841d36-kube-api-access-vhtvq\") pod \"maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") " pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:35.935891 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:35.935865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:36.052573 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.052547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:36.190460 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.190376    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"]
Jul 07 13:42:36.194676 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:42:36.194651    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9e1173_96a0_410e_a363_6bcab0841d36.slice/crio-7ee896822fa160b774abd3b223b123134324a474346bec9f1063a21d986798f8 WatchSource:0}: Error finding container 7ee896822fa160b774abd3b223b123134324a474346bec9f1063a21d986798f8: Status 404 returned error can't find the container with id 7ee896822fa160b774abd3b223b123134324a474346bec9f1063a21d986798f8
Jul 07 13:42:36.322818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.322785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:36.423461 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.423425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:36.857560 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.857507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272" event={"ID":"da9e1173-96a0-410e-a363-6bcab0841d36","Type":"ContainerStarted","Data":"2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34"}
Jul 07 13:42:36.858003 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.857631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272" event={"ID":"da9e1173-96a0-410e-a363-6bcab0841d36","Type":"ContainerStarted","Data":"7ee896822fa160b774abd3b223b123134324a474346bec9f1063a21d986798f8"}
Jul 07 13:42:36.858003 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.857759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:36.879645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:36.879559    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272" podStartSLOduration=1.460419092 podStartE2EDuration="1.879544497s" totalImagesPullingTime="419.125405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:42:36.877907364 +0000 UTC m=+1562.758278685" watchObservedRunningTime="2026-07-07 13:42:36.879544497 +0000 UTC m=+1562.759915792"
Jul 07 13:42:37.950209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:37.950177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:40.870777 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:40.870745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:41.389205 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.389153    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"]
Jul 07 13:42:41.389410 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.389387    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272" podUID="da9e1173-96a0-410e-a363-6bcab0841d36" containerName="maas-api" containerID="cri-o://2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34" gracePeriod=30
Jul 07 13:42:41.394866 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.394845    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:41.425802 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.425776    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:41.627120 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.627099    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:41.765638 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.765607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9e1173-96a0-410e-a363-6bcab0841d36-maas-api-tls\") pod \"da9e1173-96a0-410e-a363-6bcab0841d36\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") "
Jul 07 13:42:41.765803 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.765647    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhtvq\" (UniqueName: \"kubernetes.io/projected/da9e1173-96a0-410e-a363-6bcab0841d36-kube-api-access-vhtvq\") pod \"da9e1173-96a0-410e-a363-6bcab0841d36\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") "
Jul 07 13:42:41.765803 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.765702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9e1173-96a0-410e-a363-6bcab0841d36-openshift-service-ca\") pod \"da9e1173-96a0-410e-a363-6bcab0841d36\" (UID: \"da9e1173-96a0-410e-a363-6bcab0841d36\") "
Jul 07 13:42:41.766069 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.766047    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9e1173-96a0-410e-a363-6bcab0841d36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da9e1173-96a0-410e-a363-6bcab0841d36" (UID: "da9e1173-96a0-410e-a363-6bcab0841d36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:42:41.767736 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.767709    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9e1173-96a0-410e-a363-6bcab0841d36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da9e1173-96a0-410e-a363-6bcab0841d36" (UID: "da9e1173-96a0-410e-a363-6bcab0841d36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:42:41.767826 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.767811    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9e1173-96a0-410e-a363-6bcab0841d36-kube-api-access-vhtvq" (OuterVolumeSpecName: "kube-api-access-vhtvq") pod "da9e1173-96a0-410e-a363-6bcab0841d36" (UID: "da9e1173-96a0-410e-a363-6bcab0841d36"). InnerVolumeSpecName "kube-api-access-vhtvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:42:41.866833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.866804    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9e1173-96a0-410e-a363-6bcab0841d36-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:41.866833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.866833    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9e1173-96a0-410e-a363-6bcab0841d36-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:41.866985 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.866849    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhtvq\" (UniqueName: \"kubernetes.io/projected/da9e1173-96a0-410e-a363-6bcab0841d36-kube-api-access-vhtvq\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:41.883455 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.883427    2581 generic.go:356] "Generic (PLEG): container finished" podID="da9e1173-96a0-410e-a363-6bcab0841d36" containerID="2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34" exitCode=0
Jul 07 13:42:41.883782 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.883468    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"
Jul 07 13:42:41.883782 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.883477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272" event={"ID":"da9e1173-96a0-410e-a363-6bcab0841d36","Type":"ContainerDied","Data":"2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34"}
Jul 07 13:42:41.883782 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.883516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272" event={"ID":"da9e1173-96a0-410e-a363-6bcab0841d36","Type":"ContainerDied","Data":"7ee896822fa160b774abd3b223b123134324a474346bec9f1063a21d986798f8"}
Jul 07 13:42:41.883782 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.883535    2581 scope.go:117] "RemoveContainer" containerID="2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34"
Jul 07 13:42:41.892094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.892077    2581 scope.go:117] "RemoveContainer" containerID="2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34"
Jul 07 13:42:41.892363 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:42:41.892345    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34\": container with ID starting with 2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34 not found: ID does not exist" containerID="2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34"
Jul 07 13:42:41.892407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.892372    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34"} err="failed to get container status \"2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34\": rpc error: code = NotFound desc = could not find container \"2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34\": container with ID starting with 2c872a905e1169fc8f63dc8ed73ee533cdf89774a22aaf0df82346c27f1c5d34 not found: ID does not exist"
Jul 07 13:42:41.906656 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.906625    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"]
Jul 07 13:42:41.910539 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:41.910519    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-ae77d6a7-5d94bb6fd7-n8272"]
Jul 07 13:42:42.760424 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:42.760392    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9e1173-96a0-410e-a363-6bcab0841d36" path="/var/lib/kubelet/pods/da9e1173-96a0-410e-a363-6bcab0841d36/volumes"
Jul 07 13:42:47.031060 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:47.031015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:47.157493 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:47.157462    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:48.663816 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:48.663781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:53.132395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:53.132352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:53.197286 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:53.197257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:53.358972 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:53.358936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:53.459639 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:53.459554    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:54.443836 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.443798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:54.517916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.517882    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"]
Jul 07 13:42:54.518285 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.518272    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9e1173-96a0-410e-a363-6bcab0841d36" containerName="maas-api"
Jul 07 13:42:54.518344 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.518287    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9e1173-96a0-410e-a363-6bcab0841d36" containerName="maas-api"
Jul 07 13:42:54.518381 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.518362    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9e1173-96a0-410e-a363-6bcab0841d36" containerName="maas-api"
Jul 07 13:42:54.520998 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.520980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.523496 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.523475    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-a4c66a8f-serving-cert\""
Jul 07 13:42:54.523619 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.523509    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:42:54.530811 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.530786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"]
Jul 07 13:42:54.532741 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.532717    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:54.569742 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.569720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.569850 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.569757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klxz\" (UniqueName: \"kubernetes.io/projected/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-kube-api-access-4klxz\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.569850 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.569806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-openshift-service-ca\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.603280 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.603256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:54.670420 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.670395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-openshift-service-ca\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.670535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.670470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.670535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.670496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klxz\" (UniqueName: \"kubernetes.io/projected/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-kube-api-access-4klxz\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.670631 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:42:54.670613    2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-a4c66a8f-serving-cert: secret "maas-api-e2e-derive-a4c66a8f-serving-cert" not found
Jul 07 13:42:54.670692 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:42:54.670680    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls podName:1db36b02-ea24-4d97-97b9-cf2d78b0c88b nodeName:}" failed. No retries permitted until 2026-07-07 13:42:55.170660409 +0000 UTC m=+1581.051031698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls") pod "maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" (UID: "1db36b02-ea24-4d97-97b9-cf2d78b0c88b") : secret "maas-api-e2e-derive-a4c66a8f-serving-cert" not found
Jul 07 13:42:54.671097 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.671081    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-openshift-service-ca\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.679118 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.679093    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klxz\" (UniqueName: \"kubernetes.io/projected/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-kube-api-access-4klxz\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:54.995059 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:54.995026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:55.092482 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:55.092454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:55.175223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:55.175196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:55.177658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:55.177631    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls\") pod \"maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") " pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:55.433490 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:55.433418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:55.569179 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:55.569131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"]
Jul 07 13:42:55.573492 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:42:55.573458    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db36b02_ea24_4d97_97b9_cf2d78b0c88b.slice/crio-f5081dba6d595a47378fb39a6fe5ef2ca616ba9f1c5fbdd48d11e5c2f9f12085 WatchSource:0}: Error finding container f5081dba6d595a47378fb39a6fe5ef2ca616ba9f1c5fbdd48d11e5c2f9f12085: Status 404 returned error can't find the container with id f5081dba6d595a47378fb39a6fe5ef2ca616ba9f1c5fbdd48d11e5c2f9f12085
Jul 07 13:42:55.947696 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:55.947664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" event={"ID":"1db36b02-ea24-4d97-97b9-cf2d78b0c88b","Type":"ContainerStarted","Data":"f5081dba6d595a47378fb39a6fe5ef2ca616ba9f1c5fbdd48d11e5c2f9f12085"}
Jul 07 13:42:56.393236 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:56.393196    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:56.953590 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:56.953555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" event={"ID":"1db36b02-ea24-4d97-97b9-cf2d78b0c88b","Type":"ContainerStarted","Data":"a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5"}
Jul 07 13:42:56.954044 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:56.953669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:56.971346 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:56.971303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" podStartSLOduration=2.529309516 podStartE2EDuration="2.971288951s" totalImagesPullingTime="441.979435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:42:56.969288351 +0000 UTC m=+1582.849659667" watchObservedRunningTime="2026-07-07 13:42:56.971288951 +0000 UTC m=+1582.851660245"
Jul 07 13:42:58.920222 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:58.920154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:59.173928 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.173845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:59.270448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.270407    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:59.371684 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.371655    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"]
Jul 07 13:42:59.371898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.371870    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" podUID="1db36b02-ea24-4d97-97b9-cf2d78b0c88b" containerName="maas-api" containerID="cri-o://a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5" gracePeriod=30
Jul 07 13:42:59.399596 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.399570    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:42:59.631149 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.631125    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:59.713140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.713114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls\") pod \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") "
Jul 07 13:42:59.713317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.713186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-openshift-service-ca\") pod \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") "
Jul 07 13:42:59.713317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.713236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4klxz\" (UniqueName: \"kubernetes.io/projected/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-kube-api-access-4klxz\") pod \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\" (UID: \"1db36b02-ea24-4d97-97b9-cf2d78b0c88b\") "
Jul 07 13:42:59.713507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.713484    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1db36b02-ea24-4d97-97b9-cf2d78b0c88b" (UID: "1db36b02-ea24-4d97-97b9-cf2d78b0c88b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:42:59.715416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.715391    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-kube-api-access-4klxz" (OuterVolumeSpecName: "kube-api-access-4klxz") pod "1db36b02-ea24-4d97-97b9-cf2d78b0c88b" (UID: "1db36b02-ea24-4d97-97b9-cf2d78b0c88b"). InnerVolumeSpecName "kube-api-access-4klxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:42:59.715518 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.715446    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1db36b02-ea24-4d97-97b9-cf2d78b0c88b" (UID: "1db36b02-ea24-4d97-97b9-cf2d78b0c88b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:42:59.814377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.814348    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klxz\" (UniqueName: \"kubernetes.io/projected/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-kube-api-access-4klxz\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:59.814377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.814376    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:59.814554 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.814466    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db36b02-ea24-4d97-97b9-cf2d78b0c88b-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:42:59.969027 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.969000    2581 generic.go:356] "Generic (PLEG): container finished" podID="1db36b02-ea24-4d97-97b9-cf2d78b0c88b" containerID="a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5" exitCode=0
Jul 07 13:42:59.969416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.969045    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"
Jul 07 13:42:59.969416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.969062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" event={"ID":"1db36b02-ea24-4d97-97b9-cf2d78b0c88b","Type":"ContainerDied","Data":"a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5"}
Jul 07 13:42:59.969416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.969103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts" event={"ID":"1db36b02-ea24-4d97-97b9-cf2d78b0c88b","Type":"ContainerDied","Data":"f5081dba6d595a47378fb39a6fe5ef2ca616ba9f1c5fbdd48d11e5c2f9f12085"}
Jul 07 13:42:59.969416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.969123    2581 scope.go:117] "RemoveContainer" containerID="a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5"
Jul 07 13:42:59.977969 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.977940    2581 scope.go:117] "RemoveContainer" containerID="a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5"
Jul 07 13:42:59.978246 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:42:59.978228    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5\": container with ID starting with a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5 not found: ID does not exist" containerID="a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5"
Jul 07 13:42:59.978300 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.978255    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5"} err="failed to get container status \"a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5\": rpc error: code = NotFound desc = could not find container \"a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5\": container with ID starting with a8cfb193f44c1f6efb35857eb09745b58c385a638262de288393db7bc3ebaac5 not found: ID does not exist"
Jul 07 13:42:59.991093 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.991068    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"]
Jul 07 13:42:59.996825 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:42:59.996806    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-a4c66a8f-7c5bcdfc8f-l97ts"]
Jul 07 13:43:00.600940 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:00.600896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:00.760004 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:00.759970    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db36b02-ea24-4d97-97b9-cf2d78b0c88b" path="/var/lib/kubelet/pods/1db36b02-ea24-4d97-97b9-cf2d78b0c88b/volumes"
Jul 07 13:43:05.161581 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:05.161539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:05.226653 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:05.226619    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:05.643972 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:05.643934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:05.743535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:05.743503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:05.843451 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:05.843421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:07.290700 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:07.290664    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:13.227243 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:13.227212    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:13.330078 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:13.330026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:13.427361 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:13.427318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:13.528019 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:13.527981    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:13.974982 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:13.974898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:39.992791 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:39.992752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:45.653336 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:45.653290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:45.752904 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:45.752871    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:43:45.851501 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:43:45.851448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:01.780010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:01.779974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:07.444797 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:07.444753    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:07.552063 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:07.552037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:07.657048 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:07.657009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:18.258281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:18.258238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:19.103999 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:19.103960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:25.053199 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:25.053149    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:25.112373 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:25.112334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:25.200833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:25.200796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:25.294898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:25.294852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:25.393038 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:25.392962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:26.159722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:26.159694    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:26.258288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:26.258260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:26.380979 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:26.380949    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:28.066700 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:28.066661    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:33.857871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:33.857833    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:33.920846 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:33.920816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:34.570091 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:34.570056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:35.545414 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:35.545385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:40.543297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:40.543259    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:40.603526 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:40.603503    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:40.818006 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:40.817927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:41.142571 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:41.142487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:41.391553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:41.391520    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:44.115619 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.115579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:44.173789 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.173763    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"]
Jul 07 13:44:44.174203 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.174154    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db36b02-ea24-4d97-97b9-cf2d78b0c88b" containerName="maas-api"
Jul 07 13:44:44.174203 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.174191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db36b02-ea24-4d97-97b9-cf2d78b0c88b" containerName="maas-api"
Jul 07 13:44:44.174344 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.174276    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db36b02-ea24-4d97-97b9-cf2d78b0c88b" containerName="maas-api"
Jul 07 13:44:44.176667 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.176647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.179049 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.179033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:44:44.179230 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.179212    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-ba6ae1a8-serving-cert\""
Jul 07 13:44:44.189188 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.189141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"]
Jul 07 13:44:44.214138 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.214117    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:44.277750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.277722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.277848 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.277813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjzv\" (UniqueName: \"kubernetes.io/projected/c80ae02c-aaf0-4213-88fd-4860218b860a-kube-api-access-gpjzv\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.277897 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.277853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80ae02c-aaf0-4213-88fd-4860218b860a-openshift-service-ca\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.317935 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.317909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:44.378938 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.378880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjzv\" (UniqueName: \"kubernetes.io/projected/c80ae02c-aaf0-4213-88fd-4860218b860a-kube-api-access-gpjzv\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.378938 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.378927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80ae02c-aaf0-4213-88fd-4860218b860a-openshift-service-ca\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.379075 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.378989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.379139 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:44:44.379125    2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-ba6ae1a8-serving-cert: secret "maas-api-e2e-mt-ba6ae1a8-serving-cert" not found
Jul 07 13:44:44.379227 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:44:44.379209    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls podName:c80ae02c-aaf0-4213-88fd-4860218b860a nodeName:}" failed. No retries permitted until 2026-07-07 13:44:44.87919013 +0000 UTC m=+1690.759561413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls") pod "maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" (UID: "c80ae02c-aaf0-4213-88fd-4860218b860a") : secret "maas-api-e2e-mt-ba6ae1a8-serving-cert" not found
Jul 07 13:44:44.379620 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.379601    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80ae02c-aaf0-4213-88fd-4860218b860a-openshift-service-ca\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.391129 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.391109    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjzv\" (UniqueName: \"kubernetes.io/projected/c80ae02c-aaf0-4213-88fd-4860218b860a-kube-api-access-gpjzv\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.757023 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.756974    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:44.858530 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.858499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:44.883699 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.883672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.886141 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.886124    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls\") pod \"maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") " pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:44.958587 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:44.958559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:45.087791 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:45.087720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:45.239924 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:45.239899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"]
Jul 07 13:44:45.241684 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:44:45.241647    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80ae02c_aaf0_4213_88fd_4860218b860a.slice/crio-40ca52fdcfc7b9b6b10ef80bca1e7ae6b0c60e8abc30cc82f31ef711745f17fb WatchSource:0}: Error finding container 40ca52fdcfc7b9b6b10ef80bca1e7ae6b0c60e8abc30cc82f31ef711745f17fb: Status 404 returned error can't find the container with id 40ca52fdcfc7b9b6b10ef80bca1e7ae6b0c60e8abc30cc82f31ef711745f17fb
Jul 07 13:44:45.463360 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:45.463289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" event={"ID":"c80ae02c-aaf0-4213-88fd-4860218b860a","Type":"ContainerStarted","Data":"40ca52fdcfc7b9b6b10ef80bca1e7ae6b0c60e8abc30cc82f31ef711745f17fb"}
Jul 07 13:44:46.469331 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:46.469296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" event={"ID":"c80ae02c-aaf0-4213-88fd-4860218b860a","Type":"ContainerStarted","Data":"c108d77b2574e8c3462837f6a730e32af54334a6df4e87e7aa8d755aa7b278dc"}
Jul 07 13:44:46.469692 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:46.469448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:46.490696 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:46.490644    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" podStartSLOduration=1.969484171 podStartE2EDuration="2.490629787s" totalImagesPullingTime="521.145616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:44:46.486727247 +0000 UTC m=+1692.367098561" watchObservedRunningTime="2026-07-07 13:44:46.490629787 +0000 UTC m=+1692.371001096"
Jul 07 13:44:48.640787 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:48.640742    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:49.329311 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:49.329264    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:49.429597 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:49.429562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:49.802143 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:49.802094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:52.479560 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:52.479530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:55.591771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:55.591670    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:55.690613 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:55.690579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:56.016315 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.016281    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:56.294728 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.294650    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"]
Jul 07 13:44:56.294954 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.294928    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" podUID="c80ae02c-aaf0-4213-88fd-4860218b860a" containerName="maas-api" containerID="cri-o://c108d77b2574e8c3462837f6a730e32af54334a6df4e87e7aa8d755aa7b278dc" gracePeriod=30
Jul 07 13:44:56.318126 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.318095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:56.459209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.459178    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:44:56.515909 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.515875    2581 generic.go:356] "Generic (PLEG): container finished" podID="c80ae02c-aaf0-4213-88fd-4860218b860a" containerID="c108d77b2574e8c3462837f6a730e32af54334a6df4e87e7aa8d755aa7b278dc" exitCode=0
Jul 07 13:44:56.516056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.515934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" event={"ID":"c80ae02c-aaf0-4213-88fd-4860218b860a","Type":"ContainerDied","Data":"c108d77b2574e8c3462837f6a730e32af54334a6df4e87e7aa8d755aa7b278dc"}
Jul 07 13:44:56.545745 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.545691    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:56.575072 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.575047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls\") pod \"c80ae02c-aaf0-4213-88fd-4860218b860a\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") "
Jul 07 13:44:56.575246 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.575086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80ae02c-aaf0-4213-88fd-4860218b860a-openshift-service-ca\") pod \"c80ae02c-aaf0-4213-88fd-4860218b860a\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") "
Jul 07 13:44:56.575246 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.575111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpjzv\" (UniqueName: \"kubernetes.io/projected/c80ae02c-aaf0-4213-88fd-4860218b860a-kube-api-access-gpjzv\") pod \"c80ae02c-aaf0-4213-88fd-4860218b860a\" (UID: \"c80ae02c-aaf0-4213-88fd-4860218b860a\") "
Jul 07 13:44:56.575511 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.575485    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80ae02c-aaf0-4213-88fd-4860218b860a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c80ae02c-aaf0-4213-88fd-4860218b860a" (UID: "c80ae02c-aaf0-4213-88fd-4860218b860a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:44:56.577321 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.577295    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80ae02c-aaf0-4213-88fd-4860218b860a-kube-api-access-gpjzv" (OuterVolumeSpecName: "kube-api-access-gpjzv") pod "c80ae02c-aaf0-4213-88fd-4860218b860a" (UID: "c80ae02c-aaf0-4213-88fd-4860218b860a"). InnerVolumeSpecName "kube-api-access-gpjzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:44:56.577425 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.577337    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c80ae02c-aaf0-4213-88fd-4860218b860a" (UID: "c80ae02c-aaf0-4213-88fd-4860218b860a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:44:56.676437 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.676412    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c80ae02c-aaf0-4213-88fd-4860218b860a-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:44:56.676437 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.676435    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c80ae02c-aaf0-4213-88fd-4860218b860a-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:44:56.676760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:56.676444    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpjzv\" (UniqueName: \"kubernetes.io/projected/c80ae02c-aaf0-4213-88fd-4860218b860a-kube-api-access-gpjzv\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:44:57.522223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:57.522194    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"
Jul 07 13:44:57.522223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:57.522211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb" event={"ID":"c80ae02c-aaf0-4213-88fd-4860218b860a","Type":"ContainerDied","Data":"40ca52fdcfc7b9b6b10ef80bca1e7ae6b0c60e8abc30cc82f31ef711745f17fb"}
Jul 07 13:44:57.522440 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:57.522281    2581 scope.go:117] "RemoveContainer" containerID="c108d77b2574e8c3462837f6a730e32af54334a6df4e87e7aa8d755aa7b278dc"
Jul 07 13:44:57.538916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:57.538889    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"]
Jul 07 13:44:57.546086 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:57.546064    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-ba6ae1a8-58bf5c687-k67lb"]
Jul 07 13:44:58.760085 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:44:58.760053    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80ae02c-aaf0-4213-88fd-4860218b860a" path="/var/lib/kubelet/pods/c80ae02c-aaf0-4213-88fd-4860218b860a/volumes"
Jul 07 13:45:02.211326 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:02.211286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:02.313987 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:02.313958    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:02.496175 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:02.496123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:02.734525 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:02.734481    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:02.844517 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:02.844433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:02.955447 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:02.955413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:03.171217 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:03.171120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:08.926222 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:08.926187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:09.022985 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:09.022955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:09.122073 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:09.122041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:09.279431 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:09.279400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:15.019987 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:15.019951    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:15.129094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:15.129068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:22.016297 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:22.016266    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:22.337661 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:22.337576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:22.502096 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:22.502062    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:25.026087 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.026050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:25.132172 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.132123    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:25.262072 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.262044    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"]
Jul 07 13:45:25.262486 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.262470    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80ae02c-aaf0-4213-88fd-4860218b860a" containerName="maas-api"
Jul 07 13:45:25.262486 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.262488    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80ae02c-aaf0-4213-88fd-4860218b860a" containerName="maas-api"
Jul 07 13:45:25.262625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.262576    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80ae02c-aaf0-4213-88fd-4860218b860a" containerName="maas-api"
Jul 07 13:45:25.264580 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.264560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.268203 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.268180    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-e1603ba5-serving-cert\""
Jul 07 13:45:25.268534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.268506    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:45:25.294179 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.294112    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"]
Jul 07 13:45:25.307724 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.307699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdlz\" (UniqueName: \"kubernetes.io/projected/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-kube-api-access-rbdlz\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.307833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.307811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-maas-api-tls\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.307909 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.307891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-openshift-service-ca\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.326132 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.326105    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:25.408477 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.408450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-openshift-service-ca\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.408590 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.408495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdlz\" (UniqueName: \"kubernetes.io/projected/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-kube-api-access-rbdlz\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.408590 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.408544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-maas-api-tls\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.409078 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.409061    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-openshift-service-ca\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.410997 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.410977    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-maas-api-tls\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.417281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.417258    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdlz\" (UniqueName: \"kubernetes.io/projected/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-kube-api-access-rbdlz\") pod \"maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") " pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.425492 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.425467    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:25.575490 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.575426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:25.707687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.707658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"]
Jul 07 13:45:25.711458 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:45:25.711422    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78e2449_1d40_46a9_ab98_cb9c512ae4a3.slice/crio-82d72df281b70600b00913aea55eaffdc91722b2560ad20f083d455946555dd6 WatchSource:0}: Error finding container 82d72df281b70600b00913aea55eaffdc91722b2560ad20f083d455946555dd6: Status 404 returned error can't find the container with id 82d72df281b70600b00913aea55eaffdc91722b2560ad20f083d455946555dd6
Jul 07 13:45:25.823956 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.823922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:25.925494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:25.925409    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:26.658612 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:26.658568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df" event={"ID":"d78e2449-1d40-46a9-ab98-cb9c512ae4a3","Type":"ContainerStarted","Data":"9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f"}
Jul 07 13:45:26.658986 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:26.658664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df" event={"ID":"d78e2449-1d40-46a9-ab98-cb9c512ae4a3","Type":"ContainerStarted","Data":"82d72df281b70600b00913aea55eaffdc91722b2560ad20f083d455946555dd6"}
Jul 07 13:45:26.658986 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:26.658696    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:26.674883 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:26.674839    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df" podStartSLOduration=1.258390278 podStartE2EDuration="1.674825833s" totalImagesPullingTime="416.435555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:45:26.673390305 +0000 UTC m=+1732.553761618" watchObservedRunningTime="2026-07-07 13:45:26.674825833 +0000 UTC m=+1732.555197127"
Jul 07 13:45:29.540218 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:29.540187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:32.669108 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:32.669071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:45:34.386304 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.386267    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"]
Jul 07 13:45:34.389720 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.389698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.392384 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.392363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"kube-root-ca.crt\""
Jul 07 13:45:34.392504 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.392371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"default-dockercfg-dq4ht\""
Jul 07 13:45:34.393500 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.393481    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"openshift-service-ca.crt\""
Jul 07 13:45:34.393625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.393603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"e2e-lifecycle-model-e1603ba5-kserve-self-signed-certs\""
Jul 07 13:45:34.399844 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.399820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"]
Jul 07 13:45:34.419050 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.419026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:34.482996 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.482971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-model-cache\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.483152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.483001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-dshm\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.483152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.483024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tmp-dir\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.483152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.483045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-home\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.483152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.483119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tls-certs\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.483458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.483220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkv9h\" (UniqueName: \"kubernetes.io/projected/6ee0ee6e-b362-43d3-9a0e-4803c012825b-kube-api-access-pkv9h\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-model-cache\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-dshm\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tmp-dir\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-home\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tls-certs\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkv9h\" (UniqueName: \"kubernetes.io/projected/6ee0ee6e-b362-43d3-9a0e-4803c012825b-kube-api-access-pkv9h\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.584951 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584930    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-home\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.585009 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.584965    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-model-cache\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.585075 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.585057    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tmp-dir\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.587080 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.587057    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-dshm\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.587245 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.587227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"e2e-lifecycle-model-e1603ba5-kserve-self-signed-certs\""
Jul 07 13:45:34.592393 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.592373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"kube-root-ca.crt\""
Jul 07 13:45:34.597416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.597395    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tls-certs\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.603424 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.603406    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"openshift-service-ca.crt\""
Jul 07 13:45:34.614209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.614188    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkv9h\" (UniqueName: \"kubernetes.io/projected/6ee0ee6e-b362-43d3-9a0e-4803c012825b-kube-api-access-pkv9h\") pod \"e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") " pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.704335 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.704287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1603ba5\"/\"default-dockercfg-dq4ht\""
Jul 07 13:45:34.712221 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.712206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:34.850479 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:34.850455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"]
Jul 07 13:45:34.852443 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:45:34.852419    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ee0ee6e_b362_43d3_9a0e_4803c012825b.slice/crio-a056122c180a4b282a3d38fc2548e26bd2b521d81ba1aee4a12d156505b28dcf WatchSource:0}: Error finding container a056122c180a4b282a3d38fc2548e26bd2b521d81ba1aee4a12d156505b28dcf: Status 404 returned error can't find the container with id a056122c180a4b282a3d38fc2548e26bd2b521d81ba1aee4a12d156505b28dcf
Jul 07 13:45:35.702534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:35.702497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6" event={"ID":"6ee0ee6e-b362-43d3-9a0e-4803c012825b","Type":"ContainerStarted","Data":"be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589"}
Jul 07 13:45:35.702884 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:35.702543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6" event={"ID":"6ee0ee6e-b362-43d3-9a0e-4803c012825b","Type":"ContainerStarted","Data":"a056122c180a4b282a3d38fc2548e26bd2b521d81ba1aee4a12d156505b28dcf"}
Jul 07 13:45:35.720283 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:35.720233    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6" podStartSLOduration=1.720218555 podStartE2EDuration="1.720218555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:45:35.717571825 +0000 UTC m=+1741.597943120" watchObservedRunningTime="2026-07-07 13:45:35.720218555 +0000 UTC m=+1741.600589866"
Jul 07 13:45:44.712985 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:44.712948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:44.713376 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:44.712997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:44.725227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:44.725206    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:44.760217 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:44.760185    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:45:44.880149 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:44.880117    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:44.980526 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:44.980424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:47.082807 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:47.082769    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:45:47.183827 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:45:47.183796    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:46:07.815968 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:07.815929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:46:08.167119 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.167035    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"]
Jul 07 13:46:08.167725 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.167691    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df" podUID="d78e2449-1d40-46a9-ab98-cb9c512ae4a3" containerName="maas-api" containerID="cri-o://9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f" gracePeriod=30
Jul 07 13:46:08.186754 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.186725    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:46:08.419780 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.419731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:46:08.460917 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.460887    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-openshift-service-ca\") pod \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") "
Jul 07 13:46:08.461040 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.460923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-maas-api-tls\") pod \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") "
Jul 07 13:46:08.461040 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.460962    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbdlz\" (UniqueName: \"kubernetes.io/projected/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-kube-api-access-rbdlz\") pod \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\" (UID: \"d78e2449-1d40-46a9-ab98-cb9c512ae4a3\") "
Jul 07 13:46:08.461276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.461247    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d78e2449-1d40-46a9-ab98-cb9c512ae4a3" (UID: "d78e2449-1d40-46a9-ab98-cb9c512ae4a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:46:08.463138 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.463109    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-kube-api-access-rbdlz" (OuterVolumeSpecName: "kube-api-access-rbdlz") pod "d78e2449-1d40-46a9-ab98-cb9c512ae4a3" (UID: "d78e2449-1d40-46a9-ab98-cb9c512ae4a3"). InnerVolumeSpecName "kube-api-access-rbdlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:46:08.463241 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.463142    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d78e2449-1d40-46a9-ab98-cb9c512ae4a3" (UID: "d78e2449-1d40-46a9-ab98-cb9c512ae4a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:46:08.562267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.562240    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:08.562267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.562263    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:08.562267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.562272    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbdlz\" (UniqueName: \"kubernetes.io/projected/d78e2449-1d40-46a9-ab98-cb9c512ae4a3-kube-api-access-rbdlz\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:08.852562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.852529    2581 generic.go:356] "Generic (PLEG): container finished" podID="d78e2449-1d40-46a9-ab98-cb9c512ae4a3" containerID="9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f" exitCode=0
Jul 07 13:46:08.852911 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.852582    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"
Jul 07 13:46:08.852911 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.852599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df" event={"ID":"d78e2449-1d40-46a9-ab98-cb9c512ae4a3","Type":"ContainerDied","Data":"9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f"}
Jul 07 13:46:08.852911 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.852639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df" event={"ID":"d78e2449-1d40-46a9-ab98-cb9c512ae4a3","Type":"ContainerDied","Data":"82d72df281b70600b00913aea55eaffdc91722b2560ad20f083d455946555dd6"}
Jul 07 13:46:08.852911 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.852660    2581 scope.go:117] "RemoveContainer" containerID="9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f"
Jul 07 13:46:08.861221 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.861198    2581 scope.go:117] "RemoveContainer" containerID="9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f"
Jul 07 13:46:08.861489 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:46:08.861470    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f\": container with ID starting with 9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f not found: ID does not exist" containerID="9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f"
Jul 07 13:46:08.861564 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.861499    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f"} err="failed to get container status \"9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f\": rpc error: code = NotFound desc = could not find container \"9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f\": container with ID starting with 9719be40e38c0dd409d53fe83109b2ef3ed887452ac686556b0bb53f6923032f not found: ID does not exist"
Jul 07 13:46:08.868865 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.868840    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"]
Jul 07 13:46:08.872828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:08.872808    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-e1603ba5-7f5db7dbf5-x49df"]
Jul 07 13:46:10.759996 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:10.759960    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78e2449-1d40-46a9-ab98-cb9c512ae4a3" path="/var/lib/kubelet/pods/d78e2449-1d40-46a9-ab98-cb9c512ae4a3/volumes"
Jul 07 13:46:13.983880 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:13.983847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:46:18.965668 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:18.965635    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"]
Jul 07 13:46:19.014367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:19.014334    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:46:33.986384 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:33.986324    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6" podUID="6ee0ee6e-b362-43d3-9a0e-4803c012825b" containerName="main" containerID="cri-o://be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589" gracePeriod=45
Jul 07 13:46:34.231961 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.231939    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:46:34.385681 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385652    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-model-cache\") pod \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") "
Jul 07 13:46:34.385847 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkv9h\" (UniqueName: \"kubernetes.io/projected/6ee0ee6e-b362-43d3-9a0e-4803c012825b-kube-api-access-pkv9h\") pod \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") "
Jul 07 13:46:34.385847 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tmp-dir\") pod \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") "
Jul 07 13:46:34.385847 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-home\") pod \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") "
Jul 07 13:46:34.385847 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385820    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tls-certs\") pod \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") "
Jul 07 13:46:34.386008 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-dshm\") pod \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\" (UID: \"6ee0ee6e-b362-43d3-9a0e-4803c012825b\") "
Jul 07 13:46:34.386008 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385977    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ee0ee6e-b362-43d3-9a0e-4803c012825b" (UID: "6ee0ee6e-b362-43d3-9a0e-4803c012825b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:46:34.386116 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.385986    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ee0ee6e-b362-43d3-9a0e-4803c012825b" (UID: "6ee0ee6e-b362-43d3-9a0e-4803c012825b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:46:34.386116 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.386018    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-home" (OuterVolumeSpecName: "home") pod "6ee0ee6e-b362-43d3-9a0e-4803c012825b" (UID: "6ee0ee6e-b362-43d3-9a0e-4803c012825b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:46:34.386288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.386268    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:34.386334 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.386290    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:34.386334 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.386308    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:34.388047 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.388023    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-dshm" (OuterVolumeSpecName: "dshm") pod "6ee0ee6e-b362-43d3-9a0e-4803c012825b" (UID: "6ee0ee6e-b362-43d3-9a0e-4803c012825b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:46:34.388047 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.388030    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ee0ee6e-b362-43d3-9a0e-4803c012825b" (UID: "6ee0ee6e-b362-43d3-9a0e-4803c012825b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:46:34.388191 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.388097    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee0ee6e-b362-43d3-9a0e-4803c012825b-kube-api-access-pkv9h" (OuterVolumeSpecName: "kube-api-access-pkv9h") pod "6ee0ee6e-b362-43d3-9a0e-4803c012825b" (UID: "6ee0ee6e-b362-43d3-9a0e-4803c012825b"). InnerVolumeSpecName "kube-api-access-pkv9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:46:34.486938 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.486900    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkv9h\" (UniqueName: \"kubernetes.io/projected/6ee0ee6e-b362-43d3-9a0e-4803c012825b-kube-api-access-pkv9h\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:34.486938 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.486936    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee0ee6e-b362-43d3-9a0e-4803c012825b-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:34.487113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.486947    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee0ee6e-b362-43d3-9a0e-4803c012825b-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:46:34.974306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.974271    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ee0ee6e-b362-43d3-9a0e-4803c012825b" containerID="be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589" exitCode=0
Jul 07 13:46:34.974458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.974324    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"
Jul 07 13:46:34.974458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.974354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6" event={"ID":"6ee0ee6e-b362-43d3-9a0e-4803c012825b","Type":"ContainerDied","Data":"be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589"}
Jul 07 13:46:34.974458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.974397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6" event={"ID":"6ee0ee6e-b362-43d3-9a0e-4803c012825b","Type":"ContainerDied","Data":"a056122c180a4b282a3d38fc2548e26bd2b521d81ba1aee4a12d156505b28dcf"}
Jul 07 13:46:34.974458 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.974416    2581 scope.go:117] "RemoveContainer" containerID="be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589"
Jul 07 13:46:34.982958 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.982943    2581 scope.go:117] "RemoveContainer" containerID="be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589"
Jul 07 13:46:34.983232 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:46:34.983214    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589\": container with ID starting with be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589 not found: ID does not exist" containerID="be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589"
Jul 07 13:46:34.983295 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.983239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589"} err="failed to get container status \"be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589\": rpc error: code = NotFound desc = could not find container \"be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589\": container with ID starting with be90788cfae5d2074dd1bbbea3919df3608986bc6a7fa901869b38881cb75589 not found: ID does not exist"
Jul 07 13:46:34.990485 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.990461    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"]
Jul 07 13:46:34.993874 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:34.993852    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e1603ba5/e2e-lifecycle-model-e1603ba5-kserve-656fd6d8f9-bzlt6"]
Jul 07 13:46:36.760580 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:36.760543    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee0ee6e-b362-43d3-9a0e-4803c012825b" path="/var/lib/kubelet/pods/6ee0ee6e-b362-43d3-9a0e-4803c012825b/volumes"
Jul 07 13:46:55.307294 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:55.307252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:46:56.445364 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:46:56.445322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:01.794629 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:01.794595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:01.894688 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:01.894656    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:02.086234 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:02.086115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:02.316916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:02.316878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:03.431113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:03.431059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:03.522289 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:03.522257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:03.635045 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:03.635011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:04.462306 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:04.462269    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:04.561507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:04.561470    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:04.662895 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:04.662868    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:06.148871 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:06.148837    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:11.950704 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:11.950666    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:12.022525 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:12.022493    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:12.663993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:12.663952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:18.323863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:18.323820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:18.384296 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:18.384258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:18.467943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:18.467907    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:18.560535 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:18.560501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:29.071748 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:29.071709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:29.435910 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:29.435822    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:50.427213 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:50.427154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:56.005395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:56.005301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:56.105185 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:56.105128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:56.382689 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:56.382598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:56.731909 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:56.731826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:56.933712 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:56.933676    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:59.664575 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.664532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:59.763518 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.763480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:47:59.892349 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892310    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"]
Jul 07 13:47:59.892868 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892854    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee0ee6e-b362-43d3-9a0e-4803c012825b" containerName="main"
Jul 07 13:47:59.892922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892872    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee0ee6e-b362-43d3-9a0e-4803c012825b" containerName="main"
Jul 07 13:47:59.892922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892884    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78e2449-1d40-46a9-ab98-cb9c512ae4a3" containerName="maas-api"
Jul 07 13:47:59.892922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892892    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78e2449-1d40-46a9-ab98-cb9c512ae4a3" containerName="maas-api"
Jul 07 13:47:59.893028 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892965    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee0ee6e-b362-43d3-9a0e-4803c012825b" containerName="main"
Jul 07 13:47:59.893028 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.892980    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78e2449-1d40-46a9-ab98-cb9c512ae4a3" containerName="maas-api"
Jul 07 13:47:59.896525 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.896505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:47:59.899124 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.899095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-55216d-serving-cert\""
Jul 07 13:47:59.899279 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.899198    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:47:59.907430 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.907400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"]
Jul 07 13:47:59.935217 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:47:59.935104    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:00.005092 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.005061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac4db93-1f5f-4960-aad1-1a3f418045e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.005303 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.005105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztrm\" (UniqueName: \"kubernetes.io/projected/eac4db93-1f5f-4960-aad1-1a3f418045e8-kube-api-access-fztrm\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.005303 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.005213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4db93-1f5f-4960-aad1-1a3f418045e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.106499 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.106459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac4db93-1f5f-4960-aad1-1a3f418045e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.106499 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.106498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztrm\" (UniqueName: \"kubernetes.io/projected/eac4db93-1f5f-4960-aad1-1a3f418045e8-kube-api-access-fztrm\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.106731 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.106561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4db93-1f5f-4960-aad1-1a3f418045e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.107298 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.107279    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4db93-1f5f-4960-aad1-1a3f418045e8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.109439 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.109411    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac4db93-1f5f-4960-aad1-1a3f418045e8-maas-api-tls\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.115465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.115443    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztrm\" (UniqueName: \"kubernetes.io/projected/eac4db93-1f5f-4960-aad1-1a3f418045e8-kube-api-access-fztrm\") pod \"maas-api-e2e-inf-a-55216d-57d598796f-n264m\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") " pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.141807 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.141779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:00.209473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.209403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:00.337485 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.337439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:00.348374 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.348337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"]
Jul 07 13:48:00.350709 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:48:00.350681    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac4db93_1f5f_4960_aad1_1a3f418045e8.slice/crio-bc637c05d22a1a428c7ba5f4be8f308f2d14c85eae408d964ce6624d1d8b05ef WatchSource:0}: Error finding container bc637c05d22a1a428c7ba5f4be8f308f2d14c85eae408d964ce6624d1d8b05ef: Status 404 returned error can't find the container with id bc637c05d22a1a428c7ba5f4be8f308f2d14c85eae408d964ce6624d1d8b05ef
Jul 07 13:48:00.352267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.352245    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 13:48:00.438407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:00.438365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:01.361906 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:01.361871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m" event={"ID":"eac4db93-1f5f-4960-aad1-1a3f418045e8","Type":"ContainerStarted","Data":"70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871"}
Jul 07 13:48:01.362321 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:01.361928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m" event={"ID":"eac4db93-1f5f-4960-aad1-1a3f418045e8","Type":"ContainerStarted","Data":"bc637c05d22a1a428c7ba5f4be8f308f2d14c85eae408d964ce6624d1d8b05ef"}
Jul 07 13:48:01.362321 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:01.361963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:01.378262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:01.378217    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m" podStartSLOduration=1.882217738 podStartE2EDuration="2.37820219s" totalImagesPullingTime="495.984452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:48:01.37664926 +0000 UTC m=+1887.257020578" watchObservedRunningTime="2026-07-07 13:48:01.37820219 +0000 UTC m=+1887.258573483"
Jul 07 13:48:04.202013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:04.201973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:04.569275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:04.569232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:04.891953 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:04.891865    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:05.112055 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:05.112026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:06.543285 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:06.543247    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:07.371743 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.371715    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:48:07.705416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.705323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:07.762044 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.762003    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"]
Jul 07 13:48:07.765905 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.765884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.768554 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.768535    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-efd33e-serving-cert\""
Jul 07 13:48:07.775456 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.775436    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"]
Jul 07 13:48:07.789081 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.789059    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:07.866434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.866393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:07.875284 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.875255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb021247-d7a1-458e-9419-b1dd2c9759b0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.875399 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.875295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlqt\" (UniqueName: \"kubernetes.io/projected/bb021247-d7a1-458e-9419-b1dd2c9759b0-kube-api-access-hnlqt\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.875516 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.875489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb021247-d7a1-458e-9419-b1dd2c9759b0-maas-api-tls\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.976833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.976760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb021247-d7a1-458e-9419-b1dd2c9759b0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.976833 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.976793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlqt\" (UniqueName: \"kubernetes.io/projected/bb021247-d7a1-458e-9419-b1dd2c9759b0-kube-api-access-hnlqt\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.977002 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.976860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb021247-d7a1-458e-9419-b1dd2c9759b0-maas-api-tls\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.977597 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.977565    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb021247-d7a1-458e-9419-b1dd2c9759b0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.979434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.979407    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb021247-d7a1-458e-9419-b1dd2c9759b0-maas-api-tls\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:07.985470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:07.985448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlqt\" (UniqueName: \"kubernetes.io/projected/bb021247-d7a1-458e-9419-b1dd2c9759b0-kube-api-access-hnlqt\") pod \"maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") " pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:08.000174 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:08.000138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:08.080575 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:08.080556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:08.218439 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:08.218412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"]
Jul 07 13:48:08.220965 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:48:08.220940    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb021247_d7a1_458e_9419_b1dd2c9759b0.slice/crio-f930baf39dc71f045cd4c4099958e58e6344abe5cdb89f5b1ae2e4015c05a3b8 WatchSource:0}: Error finding container f930baf39dc71f045cd4c4099958e58e6344abe5cdb89f5b1ae2e4015c05a3b8: Status 404 returned error can't find the container with id f930baf39dc71f045cd4c4099958e58e6344abe5cdb89f5b1ae2e4015c05a3b8
Jul 07 13:48:08.330676 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:08.330644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:08.396435 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:08.396405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7" event={"ID":"bb021247-d7a1-458e-9419-b1dd2c9759b0","Type":"ContainerStarted","Data":"f930baf39dc71f045cd4c4099958e58e6344abe5cdb89f5b1ae2e4015c05a3b8"}
Jul 07 13:48:08.433173 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:08.433126    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:09.402768 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:09.402734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7" event={"ID":"bb021247-d7a1-458e-9419-b1dd2c9759b0","Type":"ContainerStarted","Data":"2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd"}
Jul 07 13:48:09.403133 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:09.402949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:09.419358 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:09.419311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7" podStartSLOduration=2.149486544 podStartE2EDuration="2.419297072s" totalImagesPullingTime="269.810528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:48:09.417770794 +0000 UTC m=+1895.298142100" watchObservedRunningTime="2026-07-07 13:48:09.419297072 +0000 UTC m=+1895.299668366"
Jul 07 13:48:12.238562 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:12.238519    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:15.412645 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:15.412618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:48:21.101870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:21.101828    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:21.199264 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:21.199232    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:24.873383 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.873354    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"]
Jul 07 13:48:24.877020 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.876998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:24.882673 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.882645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-55216d\"/\"test-model-55216d-kserve-self-signed-certs\""
Jul 07 13:48:24.882870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.882693    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-55216d\"/\"kube-root-ca.crt\""
Jul 07 13:48:24.882870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.882653    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-55216d\"/\"openshift-service-ca.crt\""
Jul 07 13:48:24.882870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.882653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-55216d\"/\"default-dockercfg-gt4dt\""
Jul 07 13:48:24.887197 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.887124    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"]
Jul 07 13:48:24.900218 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:24.900194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:25.012959 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.012930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-home\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.013097 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.012968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-dshm\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.013097 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.012991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ddbe963-5754-4909-8373-e46f3f732b0d-tls-certs\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.013097 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.013070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlktn\" (UniqueName: \"kubernetes.io/projected/5ddbe963-5754-4909-8373-e46f3f732b0d-kube-api-access-nlktn\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.013298 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.013141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-model-cache\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.013298 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.013207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-tmp-dir\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.113621 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.113597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-home\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.113764 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.113630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-dshm\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.113764 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.113679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ddbe963-5754-4909-8373-e46f3f732b0d-tls-certs\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.113764 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.113732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlktn\" (UniqueName: \"kubernetes.io/projected/5ddbe963-5754-4909-8373-e46f3f732b0d-kube-api-access-nlktn\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.113926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.113807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-model-cache\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.113926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.113847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-tmp-dir\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.114070 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.114043    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-home\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.114270 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.114216    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-model-cache\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.114359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.114288    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-tmp-dir\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.116259 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.116225    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-dshm\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.116377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.116362    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ddbe963-5754-4909-8373-e46f3f732b0d-tls-certs\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.121348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.121324    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlktn\" (UniqueName: \"kubernetes.io/projected/5ddbe963-5754-4909-8373-e46f3f732b0d-kube-api-access-nlktn\") pod \"test-model-55216d-kserve-769954cb4c-pncqh\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") " pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.189746 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.189664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:25.325948 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.325919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"]
Jul 07 13:48:25.328380 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:48:25.328343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ddbe963_5754_4909_8373_e46f3f732b0d.slice/crio-529def0be4abbf5dc0c907253a831126e9ae2e9b8be2b34ff7be48eed8cf5bc8 WatchSource:0}: Error finding container 529def0be4abbf5dc0c907253a831126e9ae2e9b8be2b34ff7be48eed8cf5bc8: Status 404 returned error can't find the container with id 529def0be4abbf5dc0c907253a831126e9ae2e9b8be2b34ff7be48eed8cf5bc8
Jul 07 13:48:25.479170 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.479085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh" event={"ID":"5ddbe963-5754-4909-8373-e46f3f732b0d","Type":"ContainerStarted","Data":"7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3"}
Jul 07 13:48:25.479170 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.479124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh" event={"ID":"5ddbe963-5754-4909-8373-e46f3f732b0d","Type":"ContainerStarted","Data":"529def0be4abbf5dc0c907253a831126e9ae2e9b8be2b34ff7be48eed8cf5bc8"}
Jul 07 13:48:25.498587 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:25.498532    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh" podStartSLOduration=1.498518138 podStartE2EDuration="1.498518138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:48:25.496413692 +0000 UTC m=+1911.376785003" watchObservedRunningTime="2026-07-07 13:48:25.498518138 +0000 UTC m=+1911.378889435"
Jul 07 13:48:29.184404 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:29.184363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:29.284996 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:29.284965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:48:35.190494 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:35.190458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:35.190913 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:35.190509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:35.203604 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:35.203581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:48:35.537349 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:48:35.537323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:49:09.387138 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.387105    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"]
Jul 07 13:49:09.390703 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.390683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.393349 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.393326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-efd33e\"/\"default-dockercfg-mnczn\""
Jul 07 13:49:09.393470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.393351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-efd33e\"/\"openshift-service-ca.crt\""
Jul 07 13:49:09.394380 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.394364    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-efd33e\"/\"test-model-efd33e-kserve-self-signed-certs\""
Jul 07 13:49:09.394482 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.394398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-efd33e\"/\"kube-root-ca.crt\""
Jul 07 13:49:09.401361 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.401339    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"]
Jul 07 13:49:09.411292 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.411263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:09.490363 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.490338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tmp-dir\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.490549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.490381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tls-certs\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.490549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.490409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-dshm\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.490549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.490441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-home\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.490549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.490455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-model-cache\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.490549 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.490477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxf25\" (UniqueName: \"kubernetes.io/projected/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-kube-api-access-lxf25\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591214 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tls-certs\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591319 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-dshm\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591319 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-home\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591319 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-model-cache\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591319 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxf25\" (UniqueName: \"kubernetes.io/projected/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-kube-api-access-lxf25\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591476 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tmp-dir\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.591753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591728    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-home\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.592007 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591762    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-model-cache\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.592007 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.591780    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tmp-dir\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.593581 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.593559    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-dshm\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.593811 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.593794    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tls-certs\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.599358 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.599336    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxf25\" (UniqueName: \"kubernetes.io/projected/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-kube-api-access-lxf25\") pod \"test-model-efd33e-kserve-ddb687cc7-pzsvg\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") " pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.700646 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.700575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:09.834729 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:09.834702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"]
Jul 07 13:49:09.837472 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:49:09.837443    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9fb8bc3_63b5_47bc_a1b7_d7f1fd88bf43.slice/crio-c63879e7ffd53081f8e15ac9a6e1ab0b0d7c7a3c30e703186ead1931261bbcd1 WatchSource:0}: Error finding container c63879e7ffd53081f8e15ac9a6e1ab0b0d7c7a3c30e703186ead1931261bbcd1: Status 404 returned error can't find the container with id c63879e7ffd53081f8e15ac9a6e1ab0b0d7c7a3c30e703186ead1931261bbcd1
Jul 07 13:49:10.692950 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:10.692918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg" event={"ID":"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43","Type":"ContainerStarted","Data":"7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7"}
Jul 07 13:49:10.693342 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:10.692956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg" event={"ID":"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43","Type":"ContainerStarted","Data":"c63879e7ffd53081f8e15ac9a6e1ab0b0d7c7a3c30e703186ead1931261bbcd1"}
Jul 07 13:49:10.709884 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:10.709838    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg" podStartSLOduration=1.709824298 podStartE2EDuration="1.709824298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:49:10.708193388 +0000 UTC m=+1956.588564682" watchObservedRunningTime="2026-07-07 13:49:10.709824298 +0000 UTC m=+1956.590195591"
Jul 07 13:49:10.881020 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:10.880988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:10.933030 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:10.933002    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:19.701275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:19.701241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:19.701275 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:19.701285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:19.714186 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:19.714139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:19.750286 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:19.750264    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:31.435468 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:31.435392    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"]
Jul 07 13:49:31.458514 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:31.458480    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:32.500799 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:32.500766    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"]
Jul 07 13:49:32.524417 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:32.524389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:32.676470 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:32.676434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:33.116803 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.116768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:33.220048 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.220019    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:33.317873 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.317842    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:33.377603 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.377535    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"]
Jul 07 13:49:33.377794 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.377771    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m" podUID="eac4db93-1f5f-4960-aad1-1a3f418045e8" containerName="maas-api" containerID="cri-o://70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871" gracePeriod=30
Jul 07 13:49:33.410227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.410198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:33.627218 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.627196    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:49:33.699217 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.699125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4db93-1f5f-4960-aad1-1a3f418045e8-openshift-service-ca\") pod \"eac4db93-1f5f-4960-aad1-1a3f418045e8\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") "
Jul 07 13:49:33.699217 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.699199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fztrm\" (UniqueName: \"kubernetes.io/projected/eac4db93-1f5f-4960-aad1-1a3f418045e8-kube-api-access-fztrm\") pod \"eac4db93-1f5f-4960-aad1-1a3f418045e8\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") "
Jul 07 13:49:33.699377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.699291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac4db93-1f5f-4960-aad1-1a3f418045e8-maas-api-tls\") pod \"eac4db93-1f5f-4960-aad1-1a3f418045e8\" (UID: \"eac4db93-1f5f-4960-aad1-1a3f418045e8\") "
Jul 07 13:49:33.699504 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.699479    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac4db93-1f5f-4960-aad1-1a3f418045e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eac4db93-1f5f-4960-aad1-1a3f418045e8" (UID: "eac4db93-1f5f-4960-aad1-1a3f418045e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:49:33.699679 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.699663    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eac4db93-1f5f-4960-aad1-1a3f418045e8-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:33.701304 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.701283    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac4db93-1f5f-4960-aad1-1a3f418045e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eac4db93-1f5f-4960-aad1-1a3f418045e8" (UID: "eac4db93-1f5f-4960-aad1-1a3f418045e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:49:33.701389 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.701369    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac4db93-1f5f-4960-aad1-1a3f418045e8-kube-api-access-fztrm" (OuterVolumeSpecName: "kube-api-access-fztrm") pod "eac4db93-1f5f-4960-aad1-1a3f418045e8" (UID: "eac4db93-1f5f-4960-aad1-1a3f418045e8"). InnerVolumeSpecName "kube-api-access-fztrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:49:33.800242 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800213    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztrm\" (UniqueName: \"kubernetes.io/projected/eac4db93-1f5f-4960-aad1-1a3f418045e8-kube-api-access-fztrm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:33.800242 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800240    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eac4db93-1f5f-4960-aad1-1a3f418045e8-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:33.800680 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800656    2581 generic.go:356] "Generic (PLEG): container finished" podID="eac4db93-1f5f-4960-aad1-1a3f418045e8" containerID="70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871" exitCode=0
Jul 07 13:49:33.800734 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m" event={"ID":"eac4db93-1f5f-4960-aad1-1a3f418045e8","Type":"ContainerDied","Data":"70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871"}
Jul 07 13:49:33.800774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800707    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"
Jul 07 13:49:33.800774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m" event={"ID":"eac4db93-1f5f-4960-aad1-1a3f418045e8","Type":"ContainerDied","Data":"bc637c05d22a1a428c7ba5f4be8f308f2d14c85eae408d964ce6624d1d8b05ef"}
Jul 07 13:49:33.800836 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.800774    2581 scope.go:117] "RemoveContainer" containerID="70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871"
Jul 07 13:49:33.809834 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.809820    2581 scope.go:117] "RemoveContainer" containerID="70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871"
Jul 07 13:49:33.810089 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:49:33.810071    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871\": container with ID starting with 70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871 not found: ID does not exist" containerID="70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871"
Jul 07 13:49:33.810151 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.810097    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871"} err="failed to get container status \"70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871\": rpc error: code = NotFound desc = could not find container \"70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871\": container with ID starting with 70530eb4bbedd971531d09724fffe82edb75bb1867232dea6fc407004cc3a871 not found: ID does not exist"
Jul 07 13:49:33.821394 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.821373    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"]
Jul 07 13:49:33.824983 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:33.824963    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-55216d-57d598796f-n264m"]
Jul 07 13:49:34.761074 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:34.761046    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac4db93-1f5f-4960-aad1-1a3f418045e8" path="/var/lib/kubelet/pods/eac4db93-1f5f-4960-aad1-1a3f418045e8/volumes"
Jul 07 13:49:38.181915 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:38.181885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:49:46.456128 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.456055    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh" podUID="5ddbe963-5754-4909-8373-e46f3f732b0d" containerName="main" containerID="cri-o://7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3" gracePeriod=45
Jul 07 13:49:46.703338 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.703314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:49:46.807507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ddbe963-5754-4909-8373-e46f3f732b0d-tls-certs\") pod \"5ddbe963-5754-4909-8373-e46f3f732b0d\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") "
Jul 07 13:49:46.807648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlktn\" (UniqueName: \"kubernetes.io/projected/5ddbe963-5754-4909-8373-e46f3f732b0d-kube-api-access-nlktn\") pod \"5ddbe963-5754-4909-8373-e46f3f732b0d\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") "
Jul 07 13:49:46.807648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-model-cache\") pod \"5ddbe963-5754-4909-8373-e46f3f732b0d\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") "
Jul 07 13:49:46.807648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-tmp-dir\") pod \"5ddbe963-5754-4909-8373-e46f3f732b0d\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") "
Jul 07 13:49:46.807648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-home\") pod \"5ddbe963-5754-4909-8373-e46f3f732b0d\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") "
Jul 07 13:49:46.807648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-dshm\") pod \"5ddbe963-5754-4909-8373-e46f3f732b0d\" (UID: \"5ddbe963-5754-4909-8373-e46f3f732b0d\") "
Jul 07 13:49:46.807942 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807877    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ddbe963-5754-4909-8373-e46f3f732b0d" (UID: "5ddbe963-5754-4909-8373-e46f3f732b0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:46.807942 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807888    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ddbe963-5754-4909-8373-e46f3f732b0d" (UID: "5ddbe963-5754-4909-8373-e46f3f732b0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:46.807942 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.807906    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-home" (OuterVolumeSpecName: "home") pod "5ddbe963-5754-4909-8373-e46f3f732b0d" (UID: "5ddbe963-5754-4909-8373-e46f3f732b0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:46.809817 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.809792    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-dshm" (OuterVolumeSpecName: "dshm") pod "5ddbe963-5754-4909-8373-e46f3f732b0d" (UID: "5ddbe963-5754-4909-8373-e46f3f732b0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:46.809916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.809856    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddbe963-5754-4909-8373-e46f3f732b0d-kube-api-access-nlktn" (OuterVolumeSpecName: "kube-api-access-nlktn") pod "5ddbe963-5754-4909-8373-e46f3f732b0d" (UID: "5ddbe963-5754-4909-8373-e46f3f732b0d"). InnerVolumeSpecName "kube-api-access-nlktn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:49:46.809960 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.809926    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ddbe963-5754-4909-8373-e46f3f732b0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ddbe963-5754-4909-8373-e46f3f732b0d" (UID: "5ddbe963-5754-4909-8373-e46f3f732b0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:49:46.861931 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.861906    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ddbe963-5754-4909-8373-e46f3f732b0d" containerID="7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3" exitCode=0
Jul 07 13:49:46.862032 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.861947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"
Jul 07 13:49:46.862032 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.861992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh" event={"ID":"5ddbe963-5754-4909-8373-e46f3f732b0d","Type":"ContainerDied","Data":"7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3"}
Jul 07 13:49:46.862179 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.862040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh" event={"ID":"5ddbe963-5754-4909-8373-e46f3f732b0d","Type":"ContainerDied","Data":"529def0be4abbf5dc0c907253a831126e9ae2e9b8be2b34ff7be48eed8cf5bc8"}
Jul 07 13:49:46.862179 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.862064    2581 scope.go:117] "RemoveContainer" containerID="7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3"
Jul 07 13:49:46.870426 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.870389    2581 scope.go:117] "RemoveContainer" containerID="7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3"
Jul 07 13:49:46.870686 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:49:46.870667    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3\": container with ID starting with 7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3 not found: ID does not exist" containerID="7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3"
Jul 07 13:49:46.870733 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.870694    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3"} err="failed to get container status \"7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3\": rpc error: code = NotFound desc = could not find container \"7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3\": container with ID starting with 7447727170f3536a426269424837168f71c971c0c32bab44fb9512b316322bb3 not found: ID does not exist"
Jul 07 13:49:46.884566 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.884537    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"]
Jul 07 13:49:46.888601 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.888582    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-55216d/test-model-55216d-kserve-769954cb4c-pncqh"]
Jul 07 13:49:46.909219 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.909201    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:46.909219 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.909221    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:46.909329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.909229    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:46.909329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.909237    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ddbe963-5754-4909-8373-e46f3f732b0d-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:46.909329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.909247    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlktn\" (UniqueName: \"kubernetes.io/projected/5ddbe963-5754-4909-8373-e46f3f732b0d-kube-api-access-nlktn\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:46.909329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:46.909256    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ddbe963-5754-4909-8373-e46f3f732b0d-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:47.522018 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.521971    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg" podUID="c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" containerName="main" containerID="cri-o://7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7" gracePeriod=45
Jul 07 13:49:47.782667 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.782614    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:47.815222 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tls-certs\") pod \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") "
Jul 07 13:49:47.815348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815293    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tmp-dir\") pod \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") "
Jul 07 13:49:47.815348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-dshm\") pod \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") "
Jul 07 13:49:47.815348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-model-cache\") pod \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") "
Jul 07 13:49:47.815348 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-home\") pod \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") "
Jul 07 13:49:47.815542 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxf25\" (UniqueName: \"kubernetes.io/projected/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-kube-api-access-lxf25\") pod \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\" (UID: \"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43\") "
Jul 07 13:49:47.815604 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815572    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" (UID: "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:47.815663 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815597    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" (UID: "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:47.815720 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815702    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:47.815785 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815720    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:47.815957 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.815935    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-home" (OuterVolumeSpecName: "home") pod "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" (UID: "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:47.817741 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.817719    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-dshm" (OuterVolumeSpecName: "dshm") pod "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" (UID: "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:49:47.817741 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.817727    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" (UID: "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:49:47.817905 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.817748    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-kube-api-access-lxf25" (OuterVolumeSpecName: "kube-api-access-lxf25") pod "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" (UID: "c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43"). InnerVolumeSpecName "kube-api-access-lxf25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:49:47.868704 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.868674    2581 generic.go:356] "Generic (PLEG): container finished" podID="c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" containerID="7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7" exitCode=0
Jul 07 13:49:47.868804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.868708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg" event={"ID":"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43","Type":"ContainerDied","Data":"7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7"}
Jul 07 13:49:47.868804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.868723    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"
Jul 07 13:49:47.868804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.868754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg" event={"ID":"c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43","Type":"ContainerDied","Data":"c63879e7ffd53081f8e15ac9a6e1ab0b0d7c7a3c30e703186ead1931261bbcd1"}
Jul 07 13:49:47.868804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.868778    2581 scope.go:117] "RemoveContainer" containerID="7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7"
Jul 07 13:49:47.883000 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.882979    2581 scope.go:117] "RemoveContainer" containerID="7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7"
Jul 07 13:49:47.883267 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:49:47.883246    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7\": container with ID starting with 7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7 not found: ID does not exist" containerID="7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7"
Jul 07 13:49:47.883337 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.883274    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7"} err="failed to get container status \"7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7\": rpc error: code = NotFound desc = could not find container \"7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7\": container with ID starting with 7e558a4716320adb865635a44ddea5483725a261639ae5d2e94b9f1e13b254b7 not found: ID does not exist"
Jul 07 13:49:47.892940 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.892920    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"]
Jul 07 13:49:47.896571 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.896550    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-efd33e/test-model-efd33e-kserve-ddb687cc7-pzsvg"]
Jul 07 13:49:47.916307 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.916289    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:47.916307 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.916307    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:47.916422 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.916318    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxf25\" (UniqueName: \"kubernetes.io/projected/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-kube-api-access-lxf25\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:47.916422 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:47.916327    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:49:48.760320 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:48.760275    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ddbe963-5754-4909-8373-e46f3f732b0d" path="/var/lib/kubelet/pods/5ddbe963-5754-4909-8373-e46f3f732b0d/volumes"
Jul 07 13:49:48.760844 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:49:48.760822    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" path="/var/lib/kubelet/pods/c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43/volumes"
Jul 07 13:50:07.984799 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:07.984760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:14.974417 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:14.974376    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:15.265885 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:15.265845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:15.550737 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:15.550649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:15.756229 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:15.756193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:16.134573 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.134537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:16.235460 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.235432    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:16.333830 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.333801    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"]
Jul 07 13:50:16.334042 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.334014    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7" podUID="bb021247-d7a1-458e-9419-b1dd2c9759b0" containerName="maas-api" containerID="cri-o://2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd" gracePeriod=30
Jul 07 13:50:16.363052 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.363016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:16.785211 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.785179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:50:16.961687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.961590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnlqt\" (UniqueName: \"kubernetes.io/projected/bb021247-d7a1-458e-9419-b1dd2c9759b0-kube-api-access-hnlqt\") pod \"bb021247-d7a1-458e-9419-b1dd2c9759b0\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") "
Jul 07 13:50:16.961852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.961710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb021247-d7a1-458e-9419-b1dd2c9759b0-maas-api-tls\") pod \"bb021247-d7a1-458e-9419-b1dd2c9759b0\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") "
Jul 07 13:50:16.961852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.961734    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb021247-d7a1-458e-9419-b1dd2c9759b0-openshift-service-ca\") pod \"bb021247-d7a1-458e-9419-b1dd2c9759b0\" (UID: \"bb021247-d7a1-458e-9419-b1dd2c9759b0\") "
Jul 07 13:50:16.962214 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.962185    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb021247-d7a1-458e-9419-b1dd2c9759b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb021247-d7a1-458e-9419-b1dd2c9759b0" (UID: "bb021247-d7a1-458e-9419-b1dd2c9759b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:50:16.963827 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.963791    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb021247-d7a1-458e-9419-b1dd2c9759b0-kube-api-access-hnlqt" (OuterVolumeSpecName: "kube-api-access-hnlqt") pod "bb021247-d7a1-458e-9419-b1dd2c9759b0" (UID: "bb021247-d7a1-458e-9419-b1dd2c9759b0"). InnerVolumeSpecName "kube-api-access-hnlqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:50:16.963928 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:16.963857    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb021247-d7a1-458e-9419-b1dd2c9759b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb021247-d7a1-458e-9419-b1dd2c9759b0" (UID: "bb021247-d7a1-458e-9419-b1dd2c9759b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:50:17.000711 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.000685    2581 generic.go:356] "Generic (PLEG): container finished" podID="bb021247-d7a1-458e-9419-b1dd2c9759b0" containerID="2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd" exitCode=0
Jul 07 13:50:17.000824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.000738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"
Jul 07 13:50:17.000824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.000765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7" event={"ID":"bb021247-d7a1-458e-9419-b1dd2c9759b0","Type":"ContainerDied","Data":"2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd"}
Jul 07 13:50:17.000902 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.000831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7" event={"ID":"bb021247-d7a1-458e-9419-b1dd2c9759b0","Type":"ContainerDied","Data":"f930baf39dc71f045cd4c4099958e58e6344abe5cdb89f5b1ae2e4015c05a3b8"}
Jul 07 13:50:17.000902 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.000853    2581 scope.go:117] "RemoveContainer" containerID="2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd"
Jul 07 13:50:17.010007 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.009991    2581 scope.go:117] "RemoveContainer" containerID="2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd"
Jul 07 13:50:17.010283 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:50:17.010263    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd\": container with ID starting with 2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd not found: ID does not exist" containerID="2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd"
Jul 07 13:50:17.010354 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.010293    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd"} err="failed to get container status \"2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd\": rpc error: code = NotFound desc = could not find container \"2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd\": container with ID starting with 2f5704c7dac38e448afe24297a0c4ccd3f3c51204bd0deb281c72ec11f92f0cd not found: ID does not exist"
Jul 07 13:50:17.021301 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.021276    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"]
Jul 07 13:50:17.026027 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.026007    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-efd33e-766ffc68bf-q95n7"]
Jul 07 13:50:17.062753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.062725    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb021247-d7a1-458e-9419-b1dd2c9759b0-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:17.062753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.062751    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb021247-d7a1-458e-9419-b1dd2c9759b0-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:17.062868 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:17.062762    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlqt\" (UniqueName: \"kubernetes.io/projected/bb021247-d7a1-458e-9419-b1dd2c9759b0-kube-api-access-hnlqt\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:18.761363 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:18.761330    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb021247-d7a1-458e-9419-b1dd2c9759b0" path="/var/lib/kubelet/pods/bb021247-d7a1-458e-9419-b1dd2c9759b0/volumes"
Jul 07 13:50:21.908632 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:21.908599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:22.015692 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:22.015659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:22.198903 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:22.198815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:22.447687 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:22.447646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:22.803860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:22.803823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:23.126916 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:23.126821    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:23.361725 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:23.361688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:26.390902 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.390870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:26.487663 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.487632    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"]
Jul 07 13:50:26.488035 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488020    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddbe963-5754-4909-8373-e46f3f732b0d" containerName="main"
Jul 07 13:50:26.488035 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488036    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddbe963-5754-4909-8373-e46f3f732b0d" containerName="main"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488053    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac4db93-1f5f-4960-aad1-1a3f418045e8" containerName="maas-api"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488058    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac4db93-1f5f-4960-aad1-1a3f418045e8" containerName="maas-api"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488064    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" containerName="main"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488070    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" containerName="main"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488077    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb021247-d7a1-458e-9419-b1dd2c9759b0" containerName="maas-api"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488082    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb021247-d7a1-458e-9419-b1dd2c9759b0" containerName="maas-api"
Jul 07 13:50:26.488152 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488149    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac4db93-1f5f-4960-aad1-1a3f418045e8" containerName="maas-api"
Jul 07 13:50:26.488407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488156    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddbe963-5754-4909-8373-e46f3f732b0d" containerName="main"
Jul 07 13:50:26.488407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488192    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9fb8bc3-63b5-47bc-a1b7-d7f1fd88bf43" containerName="main"
Jul 07 13:50:26.488407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.488201    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb021247-d7a1-458e-9419-b1dd2c9759b0" containerName="maas-api"
Jul 07 13:50:26.492595 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.492576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.496385 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.496368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:50:26.496473 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.496386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-6839ee-serving-cert\""
Jul 07 13:50:26.513398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.513364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:26.518019 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.517991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"]
Jul 07 13:50:26.620274 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.620227    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:26.637921 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.637898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83516fd4-dbaa-451a-a646-e5770c2a8745-openshift-service-ca\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.638034 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.637940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83516fd4-dbaa-451a-a646-e5770c2a8745-maas-api-tls\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.638034 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.638004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v82d\" (UniqueName: \"kubernetes.io/projected/83516fd4-dbaa-451a-a646-e5770c2a8745-kube-api-access-8v82d\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.738665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.738593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83516fd4-dbaa-451a-a646-e5770c2a8745-openshift-service-ca\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.738665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.738641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83516fd4-dbaa-451a-a646-e5770c2a8745-maas-api-tls\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.738869 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.738693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v82d\" (UniqueName: \"kubernetes.io/projected/83516fd4-dbaa-451a-a646-e5770c2a8745-kube-api-access-8v82d\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.739351 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.739328    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83516fd4-dbaa-451a-a646-e5770c2a8745-openshift-service-ca\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.739571 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.739545    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:26.741381 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.741362    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83516fd4-dbaa-451a-a646-e5770c2a8745-maas-api-tls\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.751688 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.751665    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v82d\" (UniqueName: \"kubernetes.io/projected/83516fd4-dbaa-451a-a646-e5770c2a8745-kube-api-access-8v82d\") pod \"maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") " pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.803649 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.803625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:26.950525 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:26.950494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"]
Jul 07 13:50:26.952206 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:50:26.952173    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83516fd4_dbaa_451a_a646_e5770c2a8745.slice/crio-2f14852d097fd147f5636c2789c2b856092e5cc7a13c0dfba24ea57a9a2d8367 WatchSource:0}: Error finding container 2f14852d097fd147f5636c2789c2b856092e5cc7a13c0dfba24ea57a9a2d8367: Status 404 returned error can't find the container with id 2f14852d097fd147f5636c2789c2b856092e5cc7a13c0dfba24ea57a9a2d8367
Jul 07 13:50:27.048465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:27.048434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8" event={"ID":"83516fd4-dbaa-451a-a646-e5770c2a8745","Type":"ContainerStarted","Data":"2f14852d097fd147f5636c2789c2b856092e5cc7a13c0dfba24ea57a9a2d8367"}
Jul 07 13:50:27.200768 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:27.200737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:27.340973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:27.340896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:28.061434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:28.061400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8" event={"ID":"83516fd4-dbaa-451a-a646-e5770c2a8745","Type":"ContainerStarted","Data":"6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a"}
Jul 07 13:50:28.061827 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:28.061492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:28.090424 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:28.090382    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8" podStartSLOduration=1.5395017050000002 podStartE2EDuration="2.090367791s" totalImagesPullingTime="550.866086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:50:28.088332215 +0000 UTC m=+2033.968703534" watchObservedRunningTime="2026-07-07 13:50:28.090367791 +0000 UTC m=+2033.970739084"
Jul 07 13:50:30.835316 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:30.835283    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:31.299710 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:31.299669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:31.633447 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:31.633357    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:31.748959 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:31.748921    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:34.071967 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.071940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:34.336463 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.336378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:34.427403 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.427359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:34.478701 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.478667    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"]
Jul 07 13:50:34.482307 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.482277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.487046 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.487021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-1904bd-serving-cert\""
Jul 07 13:50:34.498359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.498337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"]
Jul 07 13:50:34.502343 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.502316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ppv6\" (UniqueName: \"kubernetes.io/projected/a7ec3656-aeb4-4f01-8343-5af710220a5d-kube-api-access-8ppv6\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.502444 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.502362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ec3656-aeb4-4f01-8343-5af710220a5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.502522 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.502505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7ec3656-aeb4-4f01-8343-5af710220a5d-maas-api-tls\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.598961 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.598886    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:34.603131 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.603108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7ec3656-aeb4-4f01-8343-5af710220a5d-maas-api-tls\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.603258 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.603187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ppv6\" (UniqueName: \"kubernetes.io/projected/a7ec3656-aeb4-4f01-8343-5af710220a5d-kube-api-access-8ppv6\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.603258 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.603208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ec3656-aeb4-4f01-8343-5af710220a5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.603769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.603750    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ec3656-aeb4-4f01-8343-5af710220a5d-openshift-service-ca\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.607416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.607400    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-1904bd-serving-cert\""
Jul 07 13:50:34.616434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.616412    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7ec3656-aeb4-4f01-8343-5af710220a5d-maas-api-tls\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.622153 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.622126    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ppv6\" (UniqueName: \"kubernetes.io/projected/a7ec3656-aeb4-4f01-8343-5af710220a5d-kube-api-access-8ppv6\") pod \"maas-api-e2e-api-b-1904bd-d556d4776-plnnm\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") " pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.751985 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.751945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:34.797512 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.797487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:34.932322 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:34.932297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"]
Jul 07 13:50:34.934746 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:50:34.934717    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ec3656_aeb4_4f01_8343_5af710220a5d.slice/crio-2870963eb3de2108f8a0a6ecb761e93e0a2b4a6dda0c121cac06698e1bd63390 WatchSource:0}: Error finding container 2870963eb3de2108f8a0a6ecb761e93e0a2b4a6dda0c121cac06698e1bd63390: Status 404 returned error can't find the container with id 2870963eb3de2108f8a0a6ecb761e93e0a2b4a6dda0c121cac06698e1bd63390
Jul 07 13:50:35.087852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:35.087812    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:35.098402 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:35.098377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm" event={"ID":"a7ec3656-aeb4-4f01-8343-5af710220a5d","Type":"ContainerStarted","Data":"2870963eb3de2108f8a0a6ecb761e93e0a2b4a6dda0c121cac06698e1bd63390"}
Jul 07 13:50:35.188367 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:35.188297    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:36.104305 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:36.104274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm" event={"ID":"a7ec3656-aeb4-4f01-8343-5af710220a5d","Type":"ContainerStarted","Data":"ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798"}
Jul 07 13:50:36.104694 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:36.104338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:36.120681 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:36.120555    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm" podStartSLOduration=1.725725026 podStartE2EDuration="2.120535977s" totalImagesPullingTime="394.810951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:50:36.120112393 +0000 UTC m=+2042.000483705" watchObservedRunningTime="2026-07-07 13:50:36.120535977 +0000 UTC m=+2042.000907295"
Jul 07 13:50:38.865992 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:38.865964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:42.115277 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:42.115240    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:45.524834 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:45.524793    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:45.635889 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:45.635857    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:47.160977 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.160934    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:47.595721 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.595688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:47.700204 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.700150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:47.797633 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.797589    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:47.893142 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.893066    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"]
Jul 07 13:50:47.893360 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.893331    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8" podUID="83516fd4-dbaa-451a-a646-e5770c2a8745" containerName="maas-api" containerID="cri-o://6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a" gracePeriod=30
Jul 07 13:50:47.932108 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:47.932083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:48.142593 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.142571    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:48.163543 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.163465    2581 generic.go:356] "Generic (PLEG): container finished" podID="83516fd4-dbaa-451a-a646-e5770c2a8745" containerID="6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a" exitCode=0
Jul 07 13:50:48.163543 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.163509    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"
Jul 07 13:50:48.163943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.163507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8" event={"ID":"83516fd4-dbaa-451a-a646-e5770c2a8745","Type":"ContainerDied","Data":"6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a"}
Jul 07 13:50:48.163943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.163662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8" event={"ID":"83516fd4-dbaa-451a-a646-e5770c2a8745","Type":"ContainerDied","Data":"2f14852d097fd147f5636c2789c2b856092e5cc7a13c0dfba24ea57a9a2d8367"}
Jul 07 13:50:48.163943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.163704    2581 scope.go:117] "RemoveContainer" containerID="6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a"
Jul 07 13:50:48.172996 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.172978    2581 scope.go:117] "RemoveContainer" containerID="6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a"
Jul 07 13:50:48.173289 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:50:48.173269    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a\": container with ID starting with 6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a not found: ID does not exist" containerID="6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a"
Jul 07 13:50:48.173347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.173298    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a"} err="failed to get container status \"6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a\": rpc error: code = NotFound desc = could not find container \"6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a\": container with ID starting with 6936edde5c4c372825c10c5eb78a9cc6ab677ee8fc3107df7cdd3a9407affb6a not found: ID does not exist"
Jul 07 13:50:48.215196 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.215156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v82d\" (UniqueName: \"kubernetes.io/projected/83516fd4-dbaa-451a-a646-e5770c2a8745-kube-api-access-8v82d\") pod \"83516fd4-dbaa-451a-a646-e5770c2a8745\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") "
Jul 07 13:50:48.215319 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.215259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83516fd4-dbaa-451a-a646-e5770c2a8745-openshift-service-ca\") pod \"83516fd4-dbaa-451a-a646-e5770c2a8745\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") "
Jul 07 13:50:48.215387 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.215325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83516fd4-dbaa-451a-a646-e5770c2a8745-maas-api-tls\") pod \"83516fd4-dbaa-451a-a646-e5770c2a8745\" (UID: \"83516fd4-dbaa-451a-a646-e5770c2a8745\") "
Jul 07 13:50:48.215600 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.215580    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83516fd4-dbaa-451a-a646-e5770c2a8745-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83516fd4-dbaa-451a-a646-e5770c2a8745" (UID: "83516fd4-dbaa-451a-a646-e5770c2a8745"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:50:48.217221 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.217196    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83516fd4-dbaa-451a-a646-e5770c2a8745-kube-api-access-8v82d" (OuterVolumeSpecName: "kube-api-access-8v82d") pod "83516fd4-dbaa-451a-a646-e5770c2a8745" (UID: "83516fd4-dbaa-451a-a646-e5770c2a8745"). InnerVolumeSpecName "kube-api-access-8v82d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:50:48.217391 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.217371    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83516fd4-dbaa-451a-a646-e5770c2a8745-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83516fd4-dbaa-451a-a646-e5770c2a8745" (UID: "83516fd4-dbaa-451a-a646-e5770c2a8745"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:50:48.316539 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.316517    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83516fd4-dbaa-451a-a646-e5770c2a8745-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:48.316539 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.316536    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83516fd4-dbaa-451a-a646-e5770c2a8745-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:48.316665 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.316545    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v82d\" (UniqueName: \"kubernetes.io/projected/83516fd4-dbaa-451a-a646-e5770c2a8745-kube-api-access-8v82d\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:48.487750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.487721    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"]
Jul 07 13:50:48.491630 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.491603    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-6839ee-6c9f9cc4cc-pdtr8"]
Jul 07 13:50:48.760195 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:48.760151    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83516fd4-dbaa-451a-a646-e5770c2a8745" path="/var/lib/kubelet/pods/83516fd4-dbaa-451a-a646-e5770c2a8745/volumes"
Jul 07 13:50:53.353053 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:53.353014    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:53.414398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:53.414358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:53.633444 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:53.633366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:53.976198 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:53.976095    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:54.228096 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.228000    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:54.536471 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.536428    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:54.654670 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.654593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:54.755090 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.755048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:54.866877 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.866801    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"]
Jul 07 13:50:54.867073 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.867049    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm" podUID="a7ec3656-aeb4-4f01-8343-5af710220a5d" containerName="maas-api" containerID="cri-o://ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798" gracePeriod=30
Jul 07 13:50:54.891177 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:54.891129    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:50:55.117678 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.117621    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:55.173843 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.173814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ppv6\" (UniqueName: \"kubernetes.io/projected/a7ec3656-aeb4-4f01-8343-5af710220a5d-kube-api-access-8ppv6\") pod \"a7ec3656-aeb4-4f01-8343-5af710220a5d\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") "
Jul 07 13:50:55.173973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.173880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ec3656-aeb4-4f01-8343-5af710220a5d-openshift-service-ca\") pod \"a7ec3656-aeb4-4f01-8343-5af710220a5d\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") "
Jul 07 13:50:55.173973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.173905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7ec3656-aeb4-4f01-8343-5af710220a5d-maas-api-tls\") pod \"a7ec3656-aeb4-4f01-8343-5af710220a5d\" (UID: \"a7ec3656-aeb4-4f01-8343-5af710220a5d\") "
Jul 07 13:50:55.174234 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.174209    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ec3656-aeb4-4f01-8343-5af710220a5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7ec3656-aeb4-4f01-8343-5af710220a5d" (UID: "a7ec3656-aeb4-4f01-8343-5af710220a5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:50:55.175973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.175949    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ec3656-aeb4-4f01-8343-5af710220a5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7ec3656-aeb4-4f01-8343-5af710220a5d" (UID: "a7ec3656-aeb4-4f01-8343-5af710220a5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:50:55.176140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.176119    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ec3656-aeb4-4f01-8343-5af710220a5d-kube-api-access-8ppv6" (OuterVolumeSpecName: "kube-api-access-8ppv6") pod "a7ec3656-aeb4-4f01-8343-5af710220a5d" (UID: "a7ec3656-aeb4-4f01-8343-5af710220a5d"). InnerVolumeSpecName "kube-api-access-8ppv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:50:55.200338 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.200307    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7ec3656-aeb4-4f01-8343-5af710220a5d" containerID="ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798" exitCode=0
Jul 07 13:50:55.200451 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.200349    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"
Jul 07 13:50:55.200451 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.200389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm" event={"ID":"a7ec3656-aeb4-4f01-8343-5af710220a5d","Type":"ContainerDied","Data":"ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798"}
Jul 07 13:50:55.200577 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.200455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm" event={"ID":"a7ec3656-aeb4-4f01-8343-5af710220a5d","Type":"ContainerDied","Data":"2870963eb3de2108f8a0a6ecb761e93e0a2b4a6dda0c121cac06698e1bd63390"}
Jul 07 13:50:55.200577 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.200486    2581 scope.go:117] "RemoveContainer" containerID="ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798"
Jul 07 13:50:55.209590 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.209574    2581 scope.go:117] "RemoveContainer" containerID="ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798"
Jul 07 13:50:55.209874 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:50:55.209855    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798\": container with ID starting with ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798 not found: ID does not exist" containerID="ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798"
Jul 07 13:50:55.209918 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.209883    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798"} err="failed to get container status \"ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798\": rpc error: code = NotFound desc = could not find container \"ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798\": container with ID starting with ff234462b2a6b8fd5e9d0a2873d7245ed114b7c7dac363786a9a7c2117810798 not found: ID does not exist"
Jul 07 13:50:55.220514 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.220489    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"]
Jul 07 13:50:55.222699 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.222679    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-1904bd-d556d4776-plnnm"]
Jul 07 13:50:55.275459 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.275439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ppv6\" (UniqueName: \"kubernetes.io/projected/a7ec3656-aeb4-4f01-8343-5af710220a5d-kube-api-access-8ppv6\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:55.275459 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.275457    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ec3656-aeb4-4f01-8343-5af710220a5d-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:55.275585 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:55.275466    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7ec3656-aeb4-4f01-8343-5af710220a5d-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:50:56.761031 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:50:56.760995    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ec3656-aeb4-4f01-8343-5af710220a5d" path="/var/lib/kubelet/pods/a7ec3656-aeb4-4f01-8343-5af710220a5d/volumes"
Jul 07 13:51:00.188316 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:00.188270    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:00.288981 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:00.288953    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:00.436033 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:00.435969    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:00.666757 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:00.666727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:01.022094 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:01.022055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:01.336769 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:01.336691    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:01.585083 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:01.585046    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:04.442031 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.441995    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:04.508891 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.508852    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"]
Jul 07 13:51:04.509395 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.509379    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83516fd4-dbaa-451a-a646-e5770c2a8745" containerName="maas-api"
Jul 07 13:51:04.509482 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.509397    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83516fd4-dbaa-451a-a646-e5770c2a8745" containerName="maas-api"
Jul 07 13:51:04.509482 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.509436    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ec3656-aeb4-4f01-8343-5af710220a5d" containerName="maas-api"
Jul 07 13:51:04.509482 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.509445    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ec3656-aeb4-4f01-8343-5af710220a5d" containerName="maas-api"
Jul 07 13:51:04.509633 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.509518    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83516fd4-dbaa-451a-a646-e5770c2a8745" containerName="maas-api"
Jul 07 13:51:04.509633 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.509534    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ec3656-aeb4-4f01-8343-5af710220a5d" containerName="maas-api"
Jul 07 13:51:04.514523 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.514493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.517033 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.517009    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9r9sk\""
Jul 07 13:51:04.517145 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.517010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-820d4f-serving-cert\""
Jul 07 13:51:04.522134 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.522107    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"]
Jul 07 13:51:04.531981 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.531957    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:04.593538 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.593497    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:04.659064 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.659039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e027cd-eed5-4033-88ff-eed4fedb2bd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.659150 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.659104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxwd\" (UniqueName: \"kubernetes.io/projected/14e027cd-eed5-4033-88ff-eed4fedb2bd4-kube-api-access-hdxwd\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.659227 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.659150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.760311 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.760289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.760428 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.760381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e027cd-eed5-4033-88ff-eed4fedb2bd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.760492 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:51:04.760441    2581 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-820d4f-serving-cert: secret "maas-api-e2e-shared-a-820d4f-serving-cert" not found
Jul 07 13:51:04.760492 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.760448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxwd\" (UniqueName: \"kubernetes.io/projected/14e027cd-eed5-4033-88ff-eed4fedb2bd4-kube-api-access-hdxwd\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.760597 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:51:04.760515    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls podName:14e027cd-eed5-4033-88ff-eed4fedb2bd4 nodeName:}" failed. No retries permitted until 2026-07-07 13:51:05.260494434 +0000 UTC m=+2071.140865711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls") pod "maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" (UID: "14e027cd-eed5-4033-88ff-eed4fedb2bd4") : secret "maas-api-e2e-shared-a-820d4f-serving-cert" not found
Jul 07 13:51:04.761071 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.761046    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e027cd-eed5-4033-88ff-eed4fedb2bd4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:04.772977 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:04.772954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxwd\" (UniqueName: \"kubernetes.io/projected/14e027cd-eed5-4033-88ff-eed4fedb2bd4-kube-api-access-hdxwd\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:05.090481 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:05.090403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:05.192056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:05.192009    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:05.265013 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:05.264990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:05.267448 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:05.267428    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls\") pod \"maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") " pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:05.427508 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:05.427448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:05.571580 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:05.571219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"]
Jul 07 13:51:05.573519 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:51:05.573492    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e027cd_eed5_4033_88ff_eed4fedb2bd4.slice/crio-877905a103b70754b22fe5ab94517a06041f0f8316702bdf825b22613a5dba52 WatchSource:0}: Error finding container 877905a103b70754b22fe5ab94517a06041f0f8316702bdf825b22613a5dba52: Status 404 returned error can't find the container with id 877905a103b70754b22fe5ab94517a06041f0f8316702bdf825b22613a5dba52
Jul 07 13:51:06.259176 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:06.259126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" event={"ID":"14e027cd-eed5-4033-88ff-eed4fedb2bd4","Type":"ContainerStarted","Data":"09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef"}
Jul 07 13:51:06.259350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:06.259215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" event={"ID":"14e027cd-eed5-4033-88ff-eed4fedb2bd4","Type":"ContainerStarted","Data":"877905a103b70754b22fe5ab94517a06041f0f8316702bdf825b22613a5dba52"}
Jul 07 13:51:06.259350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:06.259238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:06.275908 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:06.275856    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" podStartSLOduration=1.781050296 podStartE2EDuration="2.275841201s" totalImagesPullingTime="494.790905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:51:06.274832467 +0000 UTC m=+2072.155203800" watchObservedRunningTime="2026-07-07 13:51:06.275841201 +0000 UTC m=+2072.156212495"
Jul 07 13:51:08.980668 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:08.980625    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:09.333191 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:09.333144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:09.643536 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:09.643438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:09.836852 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:09.836814    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:12.269102 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.269076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:51:12.523948 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.523856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:12.590690 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.590660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:12.843146 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.843062    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"]
Jul 07 13:51:12.846501 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.846477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:12.849859 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.849837    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-c9e14a-serving-cert\""
Jul 07 13:51:12.932837 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.932800    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"]
Jul 07 13:51:12.947320 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:12.947284    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:13.032317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.032288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c5bd2a6-338e-4621-a267-08bd1a0d433d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.032438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.032329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5bd2a6-338e-4621-a267-08bd1a0d433d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.032438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.032409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4d72\" (UniqueName: \"kubernetes.io/projected/0c5bd2a6-338e-4621-a267-08bd1a0d433d-kube-api-access-h4d72\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.035134 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.035110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:13.132908 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.132840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c5bd2a6-338e-4621-a267-08bd1a0d433d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.132908 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.132883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5bd2a6-338e-4621-a267-08bd1a0d433d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.133060 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.132930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4d72\" (UniqueName: \"kubernetes.io/projected/0c5bd2a6-338e-4621-a267-08bd1a0d433d-kube-api-access-h4d72\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.133576 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.133516    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5bd2a6-338e-4621-a267-08bd1a0d433d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.135089 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.135064    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c5bd2a6-338e-4621-a267-08bd1a0d433d-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.140513 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.140494    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4d72\" (UniqueName: \"kubernetes.io/projected/0c5bd2a6-338e-4621-a267-08bd1a0d433d-kube-api-access-h4d72\") pod \"maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") " pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.157359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.157339    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:13.292978 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.292949    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"]
Jul 07 13:51:13.296721 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:51:13.296695    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5bd2a6_338e_4621_a267_08bd1a0d433d.slice/crio-eab4ee1509f430fd8ce2429ac4bf0b5b347b0b431f91e0cf262c85e4e386fa9b WatchSource:0}: Error finding container eab4ee1509f430fd8ce2429ac4bf0b5b347b0b431f91e0cf262c85e4e386fa9b: Status 404 returned error can't find the container with id eab4ee1509f430fd8ce2429ac4bf0b5b347b0b431f91e0cf262c85e4e386fa9b
Jul 07 13:51:13.440676 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.440600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:13.540259 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:13.540205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:14.300255 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:14.300220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n" event={"ID":"0c5bd2a6-338e-4621-a267-08bd1a0d433d","Type":"ContainerStarted","Data":"61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d"}
Jul 07 13:51:14.300616 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:14.300271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n" event={"ID":"0c5bd2a6-338e-4621-a267-08bd1a0d433d","Type":"ContainerStarted","Data":"eab4ee1509f430fd8ce2429ac4bf0b5b347b0b431f91e0cf262c85e4e386fa9b"}
Jul 07 13:51:14.300616 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:14.300294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:14.319028 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:14.318988    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n" podStartSLOduration=1.960638372 podStartE2EDuration="2.318974309s" totalImagesPullingTime="358.335937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:51:14.318401311 +0000 UTC m=+2080.198772618" watchObservedRunningTime="2026-07-07 13:51:14.318974309 +0000 UTC m=+2080.199345584"
Jul 07 13:51:17.069145 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:17.069112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:20.310226 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:20.310198    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:51:25.752385 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:25.752345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:25.852800 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:25.852767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:31.687291 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.687260    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"]
Jul 07 13:51:31.690892 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.690876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.693552 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.693531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"kube-root-ca.crt\""
Jul 07 13:51:31.694734 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.694709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"default-dockercfg-b9gpc\""
Jul 07 13:51:31.694843 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.694713    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"auth-test-model-820d4f-kserve-self-signed-certs\""
Jul 07 13:51:31.694843 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.694787    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"openshift-service-ca.crt\""
Jul 07 13:51:31.699420 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.699373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"]
Jul 07 13:51:31.720416 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.720389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:31.798624 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.798599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tls-certs\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.798750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.798629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-model-cache\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.798750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.798668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-home\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.798750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.798691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsvv\" (UniqueName: \"kubernetes.io/projected/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-kube-api-access-8xsvv\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.798926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.798906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tmp-dir\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.798983 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.798964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-dshm\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.899590 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.899566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsvv\" (UniqueName: \"kubernetes.io/projected/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-kube-api-access-8xsvv\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.899682 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.899640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tmp-dir\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.899682 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.899671    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-dshm\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.899760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.899712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tls-certs\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.899760 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.899739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-model-cache\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.899843 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.899777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-home\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.900068 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.900048    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tmp-dir\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.900140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.900085    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-home\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.900226 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.900198    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-model-cache\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.902135 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.902111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-dshm\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.902334 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.902317    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tls-certs\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:31.907519 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:31.907502    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsvv\" (UniqueName: \"kubernetes.io/projected/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-kube-api-access-8xsvv\") pod \"auth-test-model-820d4f-kserve-b6b48ff5-mlvlv\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") " pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:32.006622 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:32.006597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:32.139867 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:32.139840    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"]
Jul 07 13:51:32.142418 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:51:32.142383    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b45ada1_db0b_43b2_8c51_7b5d65a7093d.slice/crio-c35c33c43e6fbd70c4b6e07db24a38e4801013ea322c5b3a2051626573fb3df4 WatchSource:0}: Error finding container c35c33c43e6fbd70c4b6e07db24a38e4801013ea322c5b3a2051626573fb3df4: Status 404 returned error can't find the container with id c35c33c43e6fbd70c4b6e07db24a38e4801013ea322c5b3a2051626573fb3df4
Jul 07 13:51:32.389965 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:32.389890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv" event={"ID":"7b45ada1-db0b-43b2-8c51-7b5d65a7093d","Type":"ContainerStarted","Data":"789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861"}
Jul 07 13:51:32.389965 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:32.389927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv" event={"ID":"7b45ada1-db0b-43b2-8c51-7b5d65a7093d","Type":"ContainerStarted","Data":"c35c33c43e6fbd70c4b6e07db24a38e4801013ea322c5b3a2051626573fb3df4"}
Jul 07 13:51:32.405118 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:32.405075    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv" podStartSLOduration=1.405061522 podStartE2EDuration="1.405061522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:51:32.404309454 +0000 UTC m=+2098.284680749" watchObservedRunningTime="2026-07-07 13:51:32.405061522 +0000 UTC m=+2098.285432815"
Jul 07 13:51:34.844066 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:34.844031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:34.944378 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:34.944343    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:42.007570 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:42.007535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:42.008038 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:42.007584    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:42.019897 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:42.019874    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:42.446136 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:42.446062    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:51:53.800818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.800779    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"]
Jul 07 13:51:53.804449 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.804432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:53.806962 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.806938    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"openshift-service-ca.crt\""
Jul 07 13:51:53.808079 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.808044    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"kube-root-ca.crt\""
Jul 07 13:51:53.808079 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.808059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"default-dockercfg-clh95\""
Jul 07 13:51:53.808079 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.808047    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"auth-test-model-c9e14a-kserve-self-signed-certs\""
Jul 07 13:51:53.814259 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.814239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"]
Jul 07 13:51:53.888676 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.888644    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:51:53.996343 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.996312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-home\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:53.996343 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.996347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c43e7fc-a634-4063-897e-2397611d7272-tls-certs\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:53.996546 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.996371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-model-cache\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:53.996546 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.996439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-dshm\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:53.996546 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.996480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-tmp-dir\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:53.996546 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:53.996502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzt7x\" (UniqueName: \"kubernetes.io/projected/4c43e7fc-a634-4063-897e-2397611d7272-kube-api-access-jzt7x\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.097924 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.097853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-model-cache\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.097924 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.097890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-dshm\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.097924 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.097922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-tmp-dir\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.098120 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.097941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzt7x\" (UniqueName: \"kubernetes.io/projected/4c43e7fc-a634-4063-897e-2397611d7272-kube-api-access-jzt7x\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.098120 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.097986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-home\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.098120 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.098006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c43e7fc-a634-4063-897e-2397611d7272-tls-certs\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.098285 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.098262    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-model-cache\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.098324 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.098314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-tmp-dir\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.098404 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.098383    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-home\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.100239 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.100218    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-dshm\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.100492 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.100474    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c43e7fc-a634-4063-897e-2397611d7272-tls-certs\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.106033 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.106009    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzt7x\" (UniqueName: \"kubernetes.io/projected/4c43e7fc-a634-4063-897e-2397611d7272-kube-api-access-jzt7x\") pod \"auth-test-model-c9e14a-kserve-5fb7489495-kvbjv\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") " pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.115894 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.115874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:51:54.270974 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.270954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"]
Jul 07 13:51:54.273310 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:51:54.273282    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c43e7fc_a634_4063_897e_2397611d7272.slice/crio-accabf7a013fd503971e88ca36ba7bc2830bf7eb9a2bd611168ed25015075810 WatchSource:0}: Error finding container accabf7a013fd503971e88ca36ba7bc2830bf7eb9a2bd611168ed25015075810: Status 404 returned error can't find the container with id accabf7a013fd503971e88ca36ba7bc2830bf7eb9a2bd611168ed25015075810
Jul 07 13:51:54.498359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.498313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv" event={"ID":"4c43e7fc-a634-4063-897e-2397611d7272","Type":"ContainerStarted","Data":"448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55"}
Jul 07 13:51:54.498359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.498354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv" event={"ID":"4c43e7fc-a634-4063-897e-2397611d7272","Type":"ContainerStarted","Data":"accabf7a013fd503971e88ca36ba7bc2830bf7eb9a2bd611168ed25015075810"}
Jul 07 13:51:54.515573 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:51:54.515526    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv" podStartSLOduration=1.515510752 podStartE2EDuration="1.515510752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:51:54.514506922 +0000 UTC m=+2120.394878214" watchObservedRunningTime="2026-07-07 13:51:54.515510752 +0000 UTC m=+2120.395882045"
Jul 07 13:52:04.117048 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:52:04.117015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:52:04.117632 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:52:04.117063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:52:04.129351 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:52:04.129330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:52:04.558384 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:52:04.558353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:52:25.669516 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:52:25.669433    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:53:01.508311 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:01.508268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:53:44.073617 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:44.073582    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"]
Jul 07 13:53:44.093789 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:44.093762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:53:44.965771 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:44.965737    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"]
Jul 07 13:53:44.986615 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:44.986581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:53:58.524752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:58.524563    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:53:59.093558 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.093517    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv" podUID="7b45ada1-db0b-43b2-8c51-7b5d65a7093d" containerName="main" containerID="cri-o://789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861" gracePeriod=45
Jul 07 13:53:59.343377 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.343353    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:53:59.472498 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tmp-dir\") pod \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") "
Jul 07 13:53:59.472498 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-home\") pod \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") "
Jul 07 13:53:59.472498 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xsvv\" (UniqueName: \"kubernetes.io/projected/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-kube-api-access-8xsvv\") pod \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") "
Jul 07 13:53:59.472752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tls-certs\") pod \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") "
Jul 07 13:53:59.472752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-dshm\") pod \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") "
Jul 07 13:53:59.472752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472574    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-model-cache\") pod \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\" (UID: \"7b45ada1-db0b-43b2-8c51-7b5d65a7093d\") "
Jul 07 13:53:59.472752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472716    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-home" (OuterVolumeSpecName: "home") pod "7b45ada1-db0b-43b2-8c51-7b5d65a7093d" (UID: "7b45ada1-db0b-43b2-8c51-7b5d65a7093d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:53:59.472752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472732    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b45ada1-db0b-43b2-8c51-7b5d65a7093d" (UID: "7b45ada1-db0b-43b2-8c51-7b5d65a7093d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:53:59.472989 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472947    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b45ada1-db0b-43b2-8c51-7b5d65a7093d" (UID: "7b45ada1-db0b-43b2-8c51-7b5d65a7093d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:53:59.472989 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472959    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:53:59.472989 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.472977    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:53:59.474846 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.474816    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-dshm" (OuterVolumeSpecName: "dshm") pod "7b45ada1-db0b-43b2-8c51-7b5d65a7093d" (UID: "7b45ada1-db0b-43b2-8c51-7b5d65a7093d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:53:59.474955 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.474885    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-kube-api-access-8xsvv" (OuterVolumeSpecName: "kube-api-access-8xsvv") pod "7b45ada1-db0b-43b2-8c51-7b5d65a7093d" (UID: "7b45ada1-db0b-43b2-8c51-7b5d65a7093d"). InnerVolumeSpecName "kube-api-access-8xsvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:53:59.474955 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.474883    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b45ada1-db0b-43b2-8c51-7b5d65a7093d" (UID: "7b45ada1-db0b-43b2-8c51-7b5d65a7093d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:53:59.574026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.573997    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xsvv\" (UniqueName: \"kubernetes.io/projected/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-kube-api-access-8xsvv\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:53:59.574026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.574020    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:53:59.574026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.574029    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:53:59.574434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.574037    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b45ada1-db0b-43b2-8c51-7b5d65a7093d-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:53:59.986789 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:53:59.986753    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv" podUID="4c43e7fc-a634-4063-897e-2397611d7272" containerName="main" containerID="cri-o://448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55" gracePeriod=45
Jul 07 13:54:00.082129 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.082099    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b45ada1-db0b-43b2-8c51-7b5d65a7093d" containerID="789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861" exitCode=0
Jul 07 13:54:00.082347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.082143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"
Jul 07 13:54:00.082347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.082178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv" event={"ID":"7b45ada1-db0b-43b2-8c51-7b5d65a7093d","Type":"ContainerDied","Data":"789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861"}
Jul 07 13:54:00.082347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.082217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv" event={"ID":"7b45ada1-db0b-43b2-8c51-7b5d65a7093d","Type":"ContainerDied","Data":"c35c33c43e6fbd70c4b6e07db24a38e4801013ea322c5b3a2051626573fb3df4"}
Jul 07 13:54:00.082347 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.082239    2581 scope.go:117] "RemoveContainer" containerID="789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861"
Jul 07 13:54:00.097579 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.097416    2581 scope.go:117] "RemoveContainer" containerID="789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861"
Jul 07 13:54:00.097690 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:54:00.097668    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861\": container with ID starting with 789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861 not found: ID does not exist" containerID="789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861"
Jul 07 13:54:00.097743 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.097699    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861"} err="failed to get container status \"789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861\": rpc error: code = NotFound desc = could not find container \"789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861\": container with ID starting with 789e1c6b46325d4a9ea82c05fe8a1ed5d160dea96fa82e462ac48ee32a724861 not found: ID does not exist"
Jul 07 13:54:00.103773 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.103750    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"]
Jul 07 13:54:00.106921 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.106895    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/auth-test-model-820d4f-kserve-b6b48ff5-mlvlv"]
Jul 07 13:54:00.230084 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.230062    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:54:00.278406 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-home\") pod \"4c43e7fc-a634-4063-897e-2397611d7272\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") "
Jul 07 13:54:00.278556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c43e7fc-a634-4063-897e-2397611d7272-tls-certs\") pod \"4c43e7fc-a634-4063-897e-2397611d7272\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") "
Jul 07 13:54:00.278556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzt7x\" (UniqueName: \"kubernetes.io/projected/4c43e7fc-a634-4063-897e-2397611d7272-kube-api-access-jzt7x\") pod \"4c43e7fc-a634-4063-897e-2397611d7272\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") "
Jul 07 13:54:00.278556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-model-cache\") pod \"4c43e7fc-a634-4063-897e-2397611d7272\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") "
Jul 07 13:54:00.278556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-dshm\") pod \"4c43e7fc-a634-4063-897e-2397611d7272\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") "
Jul 07 13:54:00.278556 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-tmp-dir\") pod \"4c43e7fc-a634-4063-897e-2397611d7272\" (UID: \"4c43e7fc-a634-4063-897e-2397611d7272\") "
Jul 07 13:54:00.278819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278747    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-home" (OuterVolumeSpecName: "home") pod "4c43e7fc-a634-4063-897e-2397611d7272" (UID: "4c43e7fc-a634-4063-897e-2397611d7272"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:54:00.278819 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278729    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c43e7fc-a634-4063-897e-2397611d7272" (UID: "4c43e7fc-a634-4063-897e-2397611d7272"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:54:00.278920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278811    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c43e7fc-a634-4063-897e-2397611d7272" (UID: "4c43e7fc-a634-4063-897e-2397611d7272"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:54:00.278973 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278951    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:54:00.279024 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278972    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:54:00.279024 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.278988    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:54:00.280808 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.280780    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-dshm" (OuterVolumeSpecName: "dshm") pod "4c43e7fc-a634-4063-897e-2397611d7272" (UID: "4c43e7fc-a634-4063-897e-2397611d7272"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:54:00.281140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.281120    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c43e7fc-a634-4063-897e-2397611d7272-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c43e7fc-a634-4063-897e-2397611d7272" (UID: "4c43e7fc-a634-4063-897e-2397611d7272"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:54:00.281140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.281134    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c43e7fc-a634-4063-897e-2397611d7272-kube-api-access-jzt7x" (OuterVolumeSpecName: "kube-api-access-jzt7x") pod "4c43e7fc-a634-4063-897e-2397611d7272" (UID: "4c43e7fc-a634-4063-897e-2397611d7272"). InnerVolumeSpecName "kube-api-access-jzt7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:54:00.379394 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.379372    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c43e7fc-a634-4063-897e-2397611d7272-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:54:00.379394 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.379392    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzt7x\" (UniqueName: \"kubernetes.io/projected/4c43e7fc-a634-4063-897e-2397611d7272-kube-api-access-jzt7x\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:54:00.379524 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.379402    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c43e7fc-a634-4063-897e-2397611d7272-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:54:00.760812 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:00.760777    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b45ada1-db0b-43b2-8c51-7b5d65a7093d" path="/var/lib/kubelet/pods/7b45ada1-db0b-43b2-8c51-7b5d65a7093d/volumes"
Jul 07 13:54:01.090006 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.089913    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c43e7fc-a634-4063-897e-2397611d7272" containerID="448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55" exitCode=0
Jul 07 13:54:01.090006 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.089958    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"
Jul 07 13:54:01.090006 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.089990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv" event={"ID":"4c43e7fc-a634-4063-897e-2397611d7272","Type":"ContainerDied","Data":"448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55"}
Jul 07 13:54:01.090317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.090035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv" event={"ID":"4c43e7fc-a634-4063-897e-2397611d7272","Type":"ContainerDied","Data":"accabf7a013fd503971e88ca36ba7bc2830bf7eb9a2bd611168ed25015075810"}
Jul 07 13:54:01.090317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.090058    2581 scope.go:117] "RemoveContainer" containerID="448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55"
Jul 07 13:54:01.101465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.101258    2581 scope.go:117] "RemoveContainer" containerID="448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55"
Jul 07 13:54:01.101531 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:54:01.101503    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55\": container with ID starting with 448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55 not found: ID does not exist" containerID="448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55"
Jul 07 13:54:01.101578 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.101525    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55"} err="failed to get container status \"448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55\": rpc error: code = NotFound desc = could not find container \"448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55\": container with ID starting with 448dd7a8ec7d973604eacac64099ced7a26266f004ae4f044fe06035d6154f55 not found: ID does not exist"
Jul 07 13:54:01.106364 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.106338    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"]
Jul 07 13:54:01.111240 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:01.111220    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/auth-test-model-c9e14a-kserve-5fb7489495-kvbjv"]
Jul 07 13:54:02.760350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:02.760312    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c43e7fc-a634-4063-897e-2397611d7272" path="/var/lib/kubelet/pods/4c43e7fc-a634-4063-897e-2397611d7272/volumes"
Jul 07 13:54:07.602400 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:07.602368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:54:18.485558 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.485524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"]
Jul 07 13:54:18.486105 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.486084    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b45ada1-db0b-43b2-8c51-7b5d65a7093d" containerName="main"
Jul 07 13:54:18.486209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.486107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b45ada1-db0b-43b2-8c51-7b5d65a7093d" containerName="main"
Jul 07 13:54:18.486209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.486131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c43e7fc-a634-4063-897e-2397611d7272" containerName="main"
Jul 07 13:54:18.486209 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.486141    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c43e7fc-a634-4063-897e-2397611d7272" containerName="main"
Jul 07 13:54:18.486374 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.486260    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c43e7fc-a634-4063-897e-2397611d7272" containerName="main"
Jul 07 13:54:18.486374 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.486279    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b45ada1-db0b-43b2-8c51-7b5d65a7093d" containerName="main"
Jul 07 13:54:18.489305 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.489283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.492317 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.492294    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"kube-root-ca.crt\""
Jul 07 13:54:18.493405 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.493381    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"default-dockercfg-clh95\""
Jul 07 13:54:18.493507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.493419    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"openshift-service-ca.crt\""
Jul 07 13:54:18.493507 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.493389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"sub-test-model-c9e14a-kserve-self-signed-certs\""
Jul 07 13:54:18.496912 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.496891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"]
Jul 07 13:54:18.515827 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.515804    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:54:18.636001 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.635972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt5gl\" (UniqueName: \"kubernetes.io/projected/33bc47ce-0320-4e58-9a77-29c811078ac8-kube-api-access-lt5gl\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.636114 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.636020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bc47ce-0320-4e58-9a77-29c811078ac8-tls-certs\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.636114 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.636071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-dshm\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.636219 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.636144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-model-cache\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.636219 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.636202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-home\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.636285 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.636226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-tmp-dir\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.737648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.737559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-model-cache\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.738318 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.738298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-home\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.738750 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.738727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-tmp-dir\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.738855 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.738668    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-home\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.738855 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.738811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt5gl\" (UniqueName: \"kubernetes.io/projected/33bc47ce-0320-4e58-9a77-29c811078ac8-kube-api-access-lt5gl\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.738981 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.738220    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-model-cache\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.739286 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.738863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bc47ce-0320-4e58-9a77-29c811078ac8-tls-certs\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.739286 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.739146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-dshm\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.739652 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.739497    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-tmp-dir\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.741592 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.741562    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-dshm\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.742189 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.742137    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bc47ce-0320-4e58-9a77-29c811078ac8-tls-certs\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.747020 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.746990    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt5gl\" (UniqueName: \"kubernetes.io/projected/33bc47ce-0320-4e58-9a77-29c811078ac8-kube-api-access-lt5gl\") pod \"sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") " pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.798946 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.798917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:18.935063 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:18.935032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"]
Jul 07 13:54:18.937676 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:54:18.937646    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33bc47ce_0320_4e58_9a77_29c811078ac8.slice/crio-76ae932fc57ec78059a0a23bf8f23b0dc1a5ea0d0061dee7997831b246465bd1 WatchSource:0}: Error finding container 76ae932fc57ec78059a0a23bf8f23b0dc1a5ea0d0061dee7997831b246465bd1: Status 404 returned error can't find the container with id 76ae932fc57ec78059a0a23bf8f23b0dc1a5ea0d0061dee7997831b246465bd1
Jul 07 13:54:19.182301 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:19.182266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp" event={"ID":"33bc47ce-0320-4e58-9a77-29c811078ac8","Type":"ContainerStarted","Data":"3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5"}
Jul 07 13:54:19.182301 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:19.182306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp" event={"ID":"33bc47ce-0320-4e58-9a77-29c811078ac8","Type":"ContainerStarted","Data":"76ae932fc57ec78059a0a23bf8f23b0dc1a5ea0d0061dee7997831b246465bd1"}
Jul 07 13:54:19.197401 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:19.197346    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp" podStartSLOduration=1.19732588 podStartE2EDuration="1.19732588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:54:19.197104042 +0000 UTC m=+2265.077475337" watchObservedRunningTime="2026-07-07 13:54:19.19732588 +0000 UTC m=+2265.077697178"
Jul 07 13:54:28.799114 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:28.799082    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:28.799502 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:28.799125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:28.811586 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:28.811567    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:29.242079 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:29.242050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:54:53.903329 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:53.903289    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:54:54.795324 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:54.795290    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"]
Jul 07 13:54:54.816419 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:54.816370    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:54:59.692127 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.692097    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"]
Jul 07 13:54:59.695729 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.695713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.699331 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.699294    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"openshift-service-ca.crt\""
Jul 07 13:54:59.699331 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.699310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"default-dockercfg-b9gpc\""
Jul 07 13:54:59.699527 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.699355    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"kube-root-ca.crt\""
Jul 07 13:54:59.699619 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.699602    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-820d4f\"/\"rate-test-model-820d4f-kserve-self-signed-certs\""
Jul 07 13:54:59.706452 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.706419    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"]
Jul 07 13:54:59.723340 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.723302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:54:59.787658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.787637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-model-cache\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.787804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.787697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ff75a9-1621-4578-9351-aefbd6931a12-tls-certs\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.787804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.787774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-home\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.787922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.787815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-dshm\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.787922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.787853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-tmp-dir\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.787922 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.787897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9bl\" (UniqueName: \"kubernetes.io/projected/05ff75a9-1621-4578-9351-aefbd6931a12-kube-api-access-fc9bl\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.888714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.888690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ff75a9-1621-4578-9351-aefbd6931a12-tls-certs\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.888828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.888727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-home\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.888828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.888757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-dshm\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.888939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.888883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-tmp-dir\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.888939 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.888930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9bl\" (UniqueName: \"kubernetes.io/projected/05ff75a9-1621-4578-9351-aefbd6931a12-kube-api-access-fc9bl\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.889053 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.888984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-model-cache\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.889191 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.889146    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-home\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.889276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.889256    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-tmp-dir\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.889321 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.889309    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-model-cache\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.891659 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.891634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-dshm\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.892024 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.891999    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ff75a9-1621-4578-9351-aefbd6931a12-tls-certs\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:54:59.897264 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:54:59.897246    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9bl\" (UniqueName: \"kubernetes.io/projected/05ff75a9-1621-4578-9351-aefbd6931a12-kube-api-access-fc9bl\") pod \"rate-test-model-820d4f-kserve-66d6555cb7-tpw55\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") " pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:55:00.006656 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:00.006631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:55:00.137824 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:00.137802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"]
Jul 07 13:55:00.140390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:55:00.140361    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ff75a9_1621_4578_9351_aefbd6931a12.slice/crio-8fc89a324fb495e72eee6571c43a2e7284c1766ec350f7a13c0d704606029181 WatchSource:0}: Error finding container 8fc89a324fb495e72eee6571c43a2e7284c1766ec350f7a13c0d704606029181: Status 404 returned error can't find the container with id 8fc89a324fb495e72eee6571c43a2e7284c1766ec350f7a13c0d704606029181
Jul 07 13:55:00.386633 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:00.386551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55" event={"ID":"05ff75a9-1621-4578-9351-aefbd6931a12","Type":"ContainerStarted","Data":"31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf"}
Jul 07 13:55:00.386633 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:00.386595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55" event={"ID":"05ff75a9-1621-4578-9351-aefbd6931a12","Type":"ContainerStarted","Data":"8fc89a324fb495e72eee6571c43a2e7284c1766ec350f7a13c0d704606029181"}
Jul 07 13:55:00.403450 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:00.403398    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55" podStartSLOduration=1.403357457 podStartE2EDuration="1.403357457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:55:00.402304993 +0000 UTC m=+2306.282676288" watchObservedRunningTime="2026-07-07 13:55:00.403357457 +0000 UTC m=+2306.283728750"
Jul 07 13:55:09.816000 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:09.815939    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp" podUID="33bc47ce-0320-4e58-9a77-29c811078ac8" containerName="main" containerID="cri-o://3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5" gracePeriod=45
Jul 07 13:55:10.007722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.007687    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:55:10.007863 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.007769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:55:10.020424 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.020403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:55:10.060436 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.060412    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:55:10.177223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-model-cache\") pod \"33bc47ce-0320-4e58-9a77-29c811078ac8\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") "
Jul 07 13:55:10.177223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt5gl\" (UniqueName: \"kubernetes.io/projected/33bc47ce-0320-4e58-9a77-29c811078ac8-kube-api-access-lt5gl\") pod \"33bc47ce-0320-4e58-9a77-29c811078ac8\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") "
Jul 07 13:55:10.177223 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-tmp-dir\") pod \"33bc47ce-0320-4e58-9a77-29c811078ac8\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") "
Jul 07 13:55:10.177487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-home\") pod \"33bc47ce-0320-4e58-9a77-29c811078ac8\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") "
Jul 07 13:55:10.177487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-dshm\") pod \"33bc47ce-0320-4e58-9a77-29c811078ac8\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") "
Jul 07 13:55:10.177487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bc47ce-0320-4e58-9a77-29c811078ac8-tls-certs\") pod \"33bc47ce-0320-4e58-9a77-29c811078ac8\" (UID: \"33bc47ce-0320-4e58-9a77-29c811078ac8\") "
Jul 07 13:55:10.177487 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177402    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-model-cache" (OuterVolumeSpecName: "model-cache") pod "33bc47ce-0320-4e58-9a77-29c811078ac8" (UID: "33bc47ce-0320-4e58-9a77-29c811078ac8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:55:10.177689 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177585    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-home" (OuterVolumeSpecName: "home") pod "33bc47ce-0320-4e58-9a77-29c811078ac8" (UID: "33bc47ce-0320-4e58-9a77-29c811078ac8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:55:10.177689 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177601    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33bc47ce-0320-4e58-9a77-29c811078ac8" (UID: "33bc47ce-0320-4e58-9a77-29c811078ac8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:55:10.177827 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177808    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:55:10.177890 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177831    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:55:10.177890 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.177844    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:55:10.179407 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.179384    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bc47ce-0320-4e58-9a77-29c811078ac8-kube-api-access-lt5gl" (OuterVolumeSpecName: "kube-api-access-lt5gl") pod "33bc47ce-0320-4e58-9a77-29c811078ac8" (UID: "33bc47ce-0320-4e58-9a77-29c811078ac8"). InnerVolumeSpecName "kube-api-access-lt5gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:55:10.179510 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.179405    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-dshm" (OuterVolumeSpecName: "dshm") pod "33bc47ce-0320-4e58-9a77-29c811078ac8" (UID: "33bc47ce-0320-4e58-9a77-29c811078ac8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:55:10.179595 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.179520    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bc47ce-0320-4e58-9a77-29c811078ac8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33bc47ce-0320-4e58-9a77-29c811078ac8" (UID: "33bc47ce-0320-4e58-9a77-29c811078ac8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:55:10.279262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.279235    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bc47ce-0320-4e58-9a77-29c811078ac8-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:55:10.279262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.279257    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bc47ce-0320-4e58-9a77-29c811078ac8-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:55:10.279262 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.279269    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt5gl\" (UniqueName: \"kubernetes.io/projected/33bc47ce-0320-4e58-9a77-29c811078ac8-kube-api-access-lt5gl\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:55:10.434784 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.434716    2581 generic.go:356] "Generic (PLEG): container finished" podID="33bc47ce-0320-4e58-9a77-29c811078ac8" containerID="3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5" exitCode=0
Jul 07 13:55:10.434784 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.434758    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"
Jul 07 13:55:10.435026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.434796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp" event={"ID":"33bc47ce-0320-4e58-9a77-29c811078ac8","Type":"ContainerDied","Data":"3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5"}
Jul 07 13:55:10.435026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.434833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp" event={"ID":"33bc47ce-0320-4e58-9a77-29c811078ac8","Type":"ContainerDied","Data":"76ae932fc57ec78059a0a23bf8f23b0dc1a5ea0d0061dee7997831b246465bd1"}
Jul 07 13:55:10.435026 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.434850    2581 scope.go:117] "RemoveContainer" containerID="3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5"
Jul 07 13:55:10.445385 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.445364    2581 scope.go:117] "RemoveContainer" containerID="3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5"
Jul 07 13:55:10.445642 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:55:10.445618    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5\": container with ID starting with 3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5 not found: ID does not exist" containerID="3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5"
Jul 07 13:55:10.445694 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.445651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5"} err="failed to get container status \"3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5\": rpc error: code = NotFound desc = could not find container \"3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5\": container with ID starting with 3d44abbcc2c89cbd18d230fa3a03de5caa837ebd76272bc6e891388ed6d64dd5 not found: ID does not exist"
Jul 07 13:55:10.448140 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.448125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:55:10.456925 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.456904    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"]
Jul 07 13:55:10.460046 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.460025    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/sub-test-model-c9e14a-kserve-86b8d5f697-tkvdp"]
Jul 07 13:55:10.760333 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:10.760304    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bc47ce-0320-4e58-9a77-29c811078ac8" path="/var/lib/kubelet/pods/33bc47ce-0320-4e58-9a77-29c811078ac8/volumes"
Jul 07 13:55:21.187956 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.187923    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"]
Jul 07 13:55:21.188359 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.188345    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bc47ce-0320-4e58-9a77-29c811078ac8" containerName="main"
Jul 07 13:55:21.188404 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.188361    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bc47ce-0320-4e58-9a77-29c811078ac8" containerName="main"
Jul 07 13:55:21.188438 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.188426    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bc47ce-0320-4e58-9a77-29c811078ac8" containerName="main"
Jul 07 13:55:21.192777 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.192752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.196683 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.196655    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"openshift-service-ca.crt\""
Jul 07 13:55:21.196812 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.196692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"kube-root-ca.crt\""
Jul 07 13:55:21.196873 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.196830    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"default-dockercfg-clh95\""
Jul 07 13:55:21.196960 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.196940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c9e14a\"/\"rate-test-model-c9e14a-kserve-self-signed-certs\""
Jul 07 13:55:21.199254 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.199211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"]
Jul 07 13:55:21.271434 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.271396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-dshm\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.271591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.271455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-home\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.271591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.271534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tmp-dir\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.271726 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.271595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tls-certs\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.271726 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.271642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwgc\" (UniqueName: \"kubernetes.io/projected/b0d6b984-edb2-4544-aa9b-a07efdb2e989-kube-api-access-pcwgc\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.271726 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.271690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-model-cache\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.302763 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.302727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:55:21.372722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.372700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-home\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.372870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.372750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tmp-dir\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.372870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.372775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tls-certs\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.372993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.372887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwgc\" (UniqueName: \"kubernetes.io/projected/b0d6b984-edb2-4544-aa9b-a07efdb2e989-kube-api-access-pcwgc\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.372993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.372959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-model-cache\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.373099 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.373041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-dshm\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.373151 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.373125    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-home\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.373246 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.373143    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tmp-dir\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.373343 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.373327    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-model-cache\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.375175 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.375136    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-dshm\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.375471 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.375454    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tls-certs\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.381193 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.381155    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwgc\" (UniqueName: \"kubernetes.io/projected/b0d6b984-edb2-4544-aa9b-a07efdb2e989-kube-api-access-pcwgc\") pod \"rate-test-model-c9e14a-kserve-55d7448599-7hwwv\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") " pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.503650 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.503624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:21.641265 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:21.641241    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"]
Jul 07 13:55:21.643390 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:55:21.643362    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d6b984_edb2_4544_aa9b_a07efdb2e989.slice/crio-94964a451f09231bf29449d60b8b8c2457bebda164f0ec03414fc07f8bb9385f WatchSource:0}: Error finding container 94964a451f09231bf29449d60b8b8c2457bebda164f0ec03414fc07f8bb9385f: Status 404 returned error can't find the container with id 94964a451f09231bf29449d60b8b8c2457bebda164f0ec03414fc07f8bb9385f
Jul 07 13:55:22.492854 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:22.492814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv" event={"ID":"b0d6b984-edb2-4544-aa9b-a07efdb2e989","Type":"ContainerStarted","Data":"16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b"}
Jul 07 13:55:22.492854 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:22.492858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv" event={"ID":"b0d6b984-edb2-4544-aa9b-a07efdb2e989","Type":"ContainerStarted","Data":"94964a451f09231bf29449d60b8b8c2457bebda164f0ec03414fc07f8bb9385f"}
Jul 07 13:55:22.510184 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:22.510115    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv" podStartSLOduration=1.510101332 podStartE2EDuration="1.510101332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:55:22.507933046 +0000 UTC m=+2328.388304340" watchObservedRunningTime="2026-07-07 13:55:22.510101332 +0000 UTC m=+2328.390472626"
Jul 07 13:55:31.504345 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:31.504272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:31.504712 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:31.504355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:31.516844 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:31.516820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:31.547433 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:31.547414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:55:46.147872 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:46.147831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:55:46.261870 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:46.261843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:55:56.917898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:55:56.917862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:17.179944 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:17.179909    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"]
Jul 07 13:56:17.199234 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:17.199199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:18.093722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:18.093692    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"]
Jul 07 13:56:18.113801 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:18.113764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:19.352260 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:19.352223    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:19.751596 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:19.751567    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:19.849023 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:19.848970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:19.948267 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:19.948233    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"]
Jul 07 13:56:19.948548 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:19.948520    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" podUID="14e027cd-eed5-4033-88ff-eed4fedb2bd4" containerName="maas-api" containerID="cri-o://09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef" gracePeriod=30
Jul 07 13:56:19.976302 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:19.976276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:20.197283 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.197259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:56:20.288461 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.288429    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls\") pod \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") "
Jul 07 13:56:20.288635 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.288538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e027cd-eed5-4033-88ff-eed4fedb2bd4-openshift-service-ca\") pod \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") "
Jul 07 13:56:20.288635 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.288568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdxwd\" (UniqueName: \"kubernetes.io/projected/14e027cd-eed5-4033-88ff-eed4fedb2bd4-kube-api-access-hdxwd\") pod \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\" (UID: \"14e027cd-eed5-4033-88ff-eed4fedb2bd4\") "
Jul 07 13:56:20.288926 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.288902    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e027cd-eed5-4033-88ff-eed4fedb2bd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14e027cd-eed5-4033-88ff-eed4fedb2bd4" (UID: "14e027cd-eed5-4033-88ff-eed4fedb2bd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:56:20.290731 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.290706    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14e027cd-eed5-4033-88ff-eed4fedb2bd4" (UID: "14e027cd-eed5-4033-88ff-eed4fedb2bd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:56:20.290838 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.290748    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e027cd-eed5-4033-88ff-eed4fedb2bd4-kube-api-access-hdxwd" (OuterVolumeSpecName: "kube-api-access-hdxwd") pod "14e027cd-eed5-4033-88ff-eed4fedb2bd4" (UID: "14e027cd-eed5-4033-88ff-eed4fedb2bd4"). InnerVolumeSpecName "kube-api-access-hdxwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:56:20.389884 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.389821    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14e027cd-eed5-4033-88ff-eed4fedb2bd4-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:20.389884 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.389843    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e027cd-eed5-4033-88ff-eed4fedb2bd4-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:20.389884 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.389853    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxwd\" (UniqueName: \"kubernetes.io/projected/14e027cd-eed5-4033-88ff-eed4fedb2bd4-kube-api-access-hdxwd\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:20.774944 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.774917    2581 generic.go:356] "Generic (PLEG): container finished" podID="14e027cd-eed5-4033-88ff-eed4fedb2bd4" containerID="09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef" exitCode=0
Jul 07 13:56:20.775095 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.774957    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"
Jul 07 13:56:20.775095 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.774982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" event={"ID":"14e027cd-eed5-4033-88ff-eed4fedb2bd4","Type":"ContainerDied","Data":"09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef"}
Jul 07 13:56:20.775095 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.775031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz" event={"ID":"14e027cd-eed5-4033-88ff-eed4fedb2bd4","Type":"ContainerDied","Data":"877905a103b70754b22fe5ab94517a06041f0f8316702bdf825b22613a5dba52"}
Jul 07 13:56:20.775095 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.775054    2581 scope.go:117] "RemoveContainer" containerID="09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef"
Jul 07 13:56:20.785864 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.785824    2581 scope.go:117] "RemoveContainer" containerID="09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef"
Jul 07 13:56:20.786252 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:56:20.786226    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef\": container with ID starting with 09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef not found: ID does not exist" containerID="09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef"
Jul 07 13:56:20.786379 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.786257    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef"} err="failed to get container status \"09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef\": rpc error: code = NotFound desc = could not find container \"09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef\": container with ID starting with 09a278af07d5a0b3139e72426a623cc41e0167bc36ddb0d86a7d7503e126d2ef not found: ID does not exist"
Jul 07 13:56:20.792087 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.792066    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"]
Jul 07 13:56:20.795524 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:20.795507    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-820d4f-6b677b4ff6-lb4lz"]
Jul 07 13:56:22.760875 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:22.760845    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e027cd-eed5-4033-88ff-eed4fedb2bd4" path="/var/lib/kubelet/pods/14e027cd-eed5-4033-88ff-eed4fedb2bd4/volumes"
Jul 07 13:56:24.845560 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:24.845526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:32.200604 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.200562    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55" podUID="05ff75a9-1621-4578-9351-aefbd6931a12" containerName="main" containerID="cri-o://31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf" gracePeriod=45
Jul 07 13:56:32.455553 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.455498    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:56:32.595552 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ff75a9-1621-4578-9351-aefbd6931a12-tls-certs\") pod \"05ff75a9-1621-4578-9351-aefbd6931a12\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") "
Jul 07 13:56:32.595688 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595591    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-home\") pod \"05ff75a9-1621-4578-9351-aefbd6931a12\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") "
Jul 07 13:56:32.595688 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc9bl\" (UniqueName: \"kubernetes.io/projected/05ff75a9-1621-4578-9351-aefbd6931a12-kube-api-access-fc9bl\") pod \"05ff75a9-1621-4578-9351-aefbd6931a12\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") "
Jul 07 13:56:32.595774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-dshm\") pod \"05ff75a9-1621-4578-9351-aefbd6931a12\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") "
Jul 07 13:56:32.595774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-model-cache\") pod \"05ff75a9-1621-4578-9351-aefbd6931a12\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") "
Jul 07 13:56:32.595774 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595768    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-tmp-dir\") pod \"05ff75a9-1621-4578-9351-aefbd6931a12\" (UID: \"05ff75a9-1621-4578-9351-aefbd6931a12\") "
Jul 07 13:56:32.595912 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.595836    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-home" (OuterVolumeSpecName: "home") pod "05ff75a9-1621-4578-9351-aefbd6931a12" (UID: "05ff75a9-1621-4578-9351-aefbd6931a12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:32.596065 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.596002    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:32.596065 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.596034    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-model-cache" (OuterVolumeSpecName: "model-cache") pod "05ff75a9-1621-4578-9351-aefbd6931a12" (UID: "05ff75a9-1621-4578-9351-aefbd6931a12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:32.596065 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.596046    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05ff75a9-1621-4578-9351-aefbd6931a12" (UID: "05ff75a9-1621-4578-9351-aefbd6931a12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:32.597937 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.597908    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-dshm" (OuterVolumeSpecName: "dshm") pod "05ff75a9-1621-4578-9351-aefbd6931a12" (UID: "05ff75a9-1621-4578-9351-aefbd6931a12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:32.598369 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.598352    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ff75a9-1621-4578-9351-aefbd6931a12-kube-api-access-fc9bl" (OuterVolumeSpecName: "kube-api-access-fc9bl") pod "05ff75a9-1621-4578-9351-aefbd6931a12" (UID: "05ff75a9-1621-4578-9351-aefbd6931a12"). InnerVolumeSpecName "kube-api-access-fc9bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:56:32.598369 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.598351    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ff75a9-1621-4578-9351-aefbd6931a12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05ff75a9-1621-4578-9351-aefbd6931a12" (UID: "05ff75a9-1621-4578-9351-aefbd6931a12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:56:32.696851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.696817    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:32.696851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.696849    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ff75a9-1621-4578-9351-aefbd6931a12-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:32.696851 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.696859    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc9bl\" (UniqueName: \"kubernetes.io/projected/05ff75a9-1621-4578-9351-aefbd6931a12-kube-api-access-fc9bl\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:32.697021 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.696868    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:32.697021 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.696877    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ff75a9-1621-4578-9351-aefbd6931a12-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:32.837797 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.837756    2581 generic.go:356] "Generic (PLEG): container finished" podID="05ff75a9-1621-4578-9351-aefbd6931a12" containerID="31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf" exitCode=0
Jul 07 13:56:32.837920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.837804    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"
Jul 07 13:56:32.837920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.837838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55" event={"ID":"05ff75a9-1621-4578-9351-aefbd6931a12","Type":"ContainerDied","Data":"31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf"}
Jul 07 13:56:32.837920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.837883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55" event={"ID":"05ff75a9-1621-4578-9351-aefbd6931a12","Type":"ContainerDied","Data":"8fc89a324fb495e72eee6571c43a2e7284c1766ec350f7a13c0d704606029181"}
Jul 07 13:56:32.837920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.837902    2581 scope.go:117] "RemoveContainer" containerID="31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf"
Jul 07 13:56:32.846279 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.846260    2581 scope.go:117] "RemoveContainer" containerID="31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf"
Jul 07 13:56:32.846576 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:56:32.846519    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf\": container with ID starting with 31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf not found: ID does not exist" containerID="31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf"
Jul 07 13:56:32.846624 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.846589    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf"} err="failed to get container status \"31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf\": rpc error: code = NotFound desc = could not find container \"31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf\": container with ID starting with 31b164584db16ac133785c8a80813dc483721408af5dd09517621bee0b96fdbf not found: ID does not exist"
Jul 07 13:56:32.854343 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.854317    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"]
Jul 07 13:56:32.860139 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:32.860121    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-820d4f/rate-test-model-820d4f-kserve-66d6555cb7-tpw55"]
Jul 07 13:56:33.115039 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.114933    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv" podUID="b0d6b984-edb2-4544-aa9b-a07efdb2e989" containerName="main" containerID="cri-o://16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b" gracePeriod=45
Jul 07 13:56:33.374340 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.374284    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:56:33.502554 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-dshm\") pod \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") "
Jul 07 13:56:33.502714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcwgc\" (UniqueName: \"kubernetes.io/projected/b0d6b984-edb2-4544-aa9b-a07efdb2e989-kube-api-access-pcwgc\") pod \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") "
Jul 07 13:56:33.502714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-home\") pod \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") "
Jul 07 13:56:33.502714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-model-cache\") pod \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") "
Jul 07 13:56:33.502714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tmp-dir\") pod \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") "
Jul 07 13:56:33.502714 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tls-certs\") pod \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\" (UID: \"b0d6b984-edb2-4544-aa9b-a07efdb2e989\") "
Jul 07 13:56:33.502999 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502868    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-home" (OuterVolumeSpecName: "home") pod "b0d6b984-edb2-4544-aa9b-a07efdb2e989" (UID: "b0d6b984-edb2-4544-aa9b-a07efdb2e989"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:33.502999 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502897    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0d6b984-edb2-4544-aa9b-a07efdb2e989" (UID: "b0d6b984-edb2-4544-aa9b-a07efdb2e989"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:33.502999 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.502967    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0d6b984-edb2-4544-aa9b-a07efdb2e989" (UID: "b0d6b984-edb2-4544-aa9b-a07efdb2e989"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:33.503116 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.503033    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-model-cache\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:33.503116 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.503054    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-home\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:33.504753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.504732    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d6b984-edb2-4544-aa9b-a07efdb2e989-kube-api-access-pcwgc" (OuterVolumeSpecName: "kube-api-access-pcwgc") pod "b0d6b984-edb2-4544-aa9b-a07efdb2e989" (UID: "b0d6b984-edb2-4544-aa9b-a07efdb2e989"). InnerVolumeSpecName "kube-api-access-pcwgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:56:33.504753 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.504743    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-dshm" (OuterVolumeSpecName: "dshm") pod "b0d6b984-edb2-4544-aa9b-a07efdb2e989" (UID: "b0d6b984-edb2-4544-aa9b-a07efdb2e989"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 13:56:33.504920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.504901    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0d6b984-edb2-4544-aa9b-a07efdb2e989" (UID: "b0d6b984-edb2-4544-aa9b-a07efdb2e989"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:56:33.604391 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.604369    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tls-certs\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:33.604391 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.604391    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-dshm\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:33.604534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.604401    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwgc\" (UniqueName: \"kubernetes.io/projected/b0d6b984-edb2-4544-aa9b-a07efdb2e989-kube-api-access-pcwgc\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:33.604534 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.604410    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d6b984-edb2-4544-aa9b-a07efdb2e989-tmp-dir\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:56:33.845112 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.845080    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0d6b984-edb2-4544-aa9b-a07efdb2e989" containerID="16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b" exitCode=0
Jul 07 13:56:33.845288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.845125    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"
Jul 07 13:56:33.845288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.845128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv" event={"ID":"b0d6b984-edb2-4544-aa9b-a07efdb2e989","Type":"ContainerDied","Data":"16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b"}
Jul 07 13:56:33.845288 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.845264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv" event={"ID":"b0d6b984-edb2-4544-aa9b-a07efdb2e989","Type":"ContainerDied","Data":"94964a451f09231bf29449d60b8b8c2457bebda164f0ec03414fc07f8bb9385f"}
Jul 07 13:56:33.845446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.845292    2581 scope.go:117] "RemoveContainer" containerID="16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b"
Jul 07 13:56:33.854107 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.854090    2581 scope.go:117] "RemoveContainer" containerID="16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b"
Jul 07 13:56:33.854369 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:56:33.854352    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b\": container with ID starting with 16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b not found: ID does not exist" containerID="16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b"
Jul 07 13:56:33.854425 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.854380    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b"} err="failed to get container status \"16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b\": rpc error: code = NotFound desc = could not find container \"16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b\": container with ID starting with 16e280f86492022ec69c4bc702b608582c862aaa2671db21376a3702ac3fff6b not found: ID does not exist"
Jul 07 13:56:33.866544 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.866519    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"]
Jul 07 13:56:33.868122 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:33.868103    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c9e14a/rate-test-model-c9e14a-kserve-55d7448599-7hwwv"]
Jul 07 13:56:34.762148 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:34.762113    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ff75a9-1621-4578-9351-aefbd6931a12" path="/var/lib/kubelet/pods/05ff75a9-1621-4578-9351-aefbd6931a12/volumes"
Jul 07 13:56:34.762660 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:34.762643    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d6b984-edb2-4544-aa9b-a07efdb2e989" path="/var/lib/kubelet/pods/b0d6b984-edb2-4544-aa9b-a07efdb2e989/volumes"
Jul 07 13:56:35.595231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:35.595186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:56:35.697322 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:56:35.697287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:01.741073 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:01.740980    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:02.030593 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.030552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:02.268644 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.268605    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:02.425591 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.425512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:02.679820 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.679728    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"]
Jul 07 13:57:02.680060 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.680034    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n" podUID="0c5bd2a6-338e-4621-a267-08bd1a0d433d" containerName="maas-api" containerID="cri-o://61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d" gracePeriod=30
Jul 07 13:57:02.712806 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.712770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:02.859362 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.859319    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:02.932616 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.932561    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:57:02.949042 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.949012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c5bd2a6-338e-4621-a267-08bd1a0d433d-maas-api-tls\") pod \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") "
Jul 07 13:57:02.949191 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.949114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5bd2a6-338e-4621-a267-08bd1a0d433d-openshift-service-ca\") pod \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") "
Jul 07 13:57:02.949191 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.949145    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4d72\" (UniqueName: \"kubernetes.io/projected/0c5bd2a6-338e-4621-a267-08bd1a0d433d-kube-api-access-h4d72\") pod \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\" (UID: \"0c5bd2a6-338e-4621-a267-08bd1a0d433d\") "
Jul 07 13:57:02.949613 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.949588    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5bd2a6-338e-4621-a267-08bd1a0d433d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c5bd2a6-338e-4621-a267-08bd1a0d433d" (UID: "0c5bd2a6-338e-4621-a267-08bd1a0d433d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 13:57:02.951818 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.951746    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5bd2a6-338e-4621-a267-08bd1a0d433d-kube-api-access-h4d72" (OuterVolumeSpecName: "kube-api-access-h4d72") pod "0c5bd2a6-338e-4621-a267-08bd1a0d433d" (UID: "0c5bd2a6-338e-4621-a267-08bd1a0d433d"). InnerVolumeSpecName "kube-api-access-h4d72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 13:57:02.952021 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.951999    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5bd2a6-338e-4621-a267-08bd1a0d433d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c5bd2a6-338e-4621-a267-08bd1a0d433d" (UID: "0c5bd2a6-338e-4621-a267-08bd1a0d433d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 13:57:02.983630 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.983603    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c5bd2a6-338e-4621-a267-08bd1a0d433d" containerID="61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d" exitCode=0
Jul 07 13:57:02.983735 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.983650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"
Jul 07 13:57:02.983735 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.983668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n" event={"ID":"0c5bd2a6-338e-4621-a267-08bd1a0d433d","Type":"ContainerDied","Data":"61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d"}
Jul 07 13:57:02.983735 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.983722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n" event={"ID":"0c5bd2a6-338e-4621-a267-08bd1a0d433d","Type":"ContainerDied","Data":"eab4ee1509f430fd8ce2429ac4bf0b5b347b0b431f91e0cf262c85e4e386fa9b"}
Jul 07 13:57:02.983898 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.983751    2581 scope.go:117] "RemoveContainer" containerID="61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d"
Jul 07 13:57:02.993222 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.993212    2581 scope.go:117] "RemoveContainer" containerID="61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d"
Jul 07 13:57:02.993479 ip-10-0-133-157 kubenswrapper[2581]: E0707 13:57:02.993458    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d\": container with ID starting with 61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d not found: ID does not exist" containerID="61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d"
Jul 07 13:57:02.993551 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:02.993491    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d"} err="failed to get container status \"61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d\": rpc error: code = NotFound desc = could not find container \"61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d\": container with ID starting with 61c9601846e1ad85017cbc20069ca356ea3cb0fa8fa2e67b1c5da5f3e5ed580d not found: ID does not exist"
Jul 07 13:57:03.004492 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:03.004467    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"]
Jul 07 13:57:03.007944 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:03.007920    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c9e14a-57b9b5f9c8-q7k5n"]
Jul 07 13:57:03.049886 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:03.049861    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c5bd2a6-338e-4621-a267-08bd1a0d433d-maas-api-tls\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:57:03.049886 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:03.049885    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c5bd2a6-338e-4621-a267-08bd1a0d433d-openshift-service-ca\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:57:03.050010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:03.049901    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4d72\" (UniqueName: \"kubernetes.io/projected/0c5bd2a6-338e-4621-a267-08bd1a0d433d-kube-api-access-h4d72\") on node \"ip-10-0-133-157.ec2.internal\" DevicePath \"\""
Jul 07 13:57:04.761885 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:04.761850    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c5bd2a6-338e-4621-a267-08bd1a0d433d" path="/var/lib/kubelet/pods/0c5bd2a6-338e-4621-a267-08bd1a0d433d/volumes"
Jul 07 13:57:08.706670 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:08.706626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:08.771533 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:08.771506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:08.984104 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:08.984011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:09.206781 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:09.206749    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:15.571397 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:15.571368    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fbc88fbc-q5ghb_0522fb53-79eb-47f6-ac7a-2553b6d7c868/copy-manifests/0.log"
Jul 07 13:57:15.577584 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:15.577554    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fbc88fbc-q5ghb_0522fb53-79eb-47f6-ac7a-2553b6d7c868/manager/0.log"
Jul 07 13:57:17.742214 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:17.742183    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-tsdhk_13d458d1-fdf4-4a80-9666-0e5b949c93f3/kuadrant-console-plugin/0.log"
Jul 07 13:57:18.238524 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:18.238446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dp5l6_1a1140db-844d-4c86-ba2b-47264b20f075/manager/0.log"
Jul 07 13:57:18.959514 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:18.959483    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577c5f8c7-qxgnv_5540609e-e8e9-483f-8ce9-57ff0fc14da7/kube-auth-proxy/0.log"
Jul 07 13:57:19.220454 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:19.220384    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-4nrs6_288a9386-9eb8-4146-9d36-1f1da3098d34/payload-pre-processing/0.log"
Jul 07 13:57:19.946648 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:19.946614    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9xdkv_6623c51c-1307-4425-9966-4a6f11ef005b/main/0.log"
Jul 07 13:57:20.428446 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:20.428411    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-8lw4c_a98a46bf-1d6d-47c2-9259-fed852d7c1fe/main/0.log"
Jul 07 13:57:25.275763 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.275733    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4dq5n/must-gather-cdwbk"]
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276066    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5bd2a6-338e-4621-a267-08bd1a0d433d" containerName="maas-api"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5bd2a6-338e-4621-a267-08bd1a0d433d" containerName="maas-api"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e027cd-eed5-4033-88ff-eed4fedb2bd4" containerName="maas-api"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276094    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e027cd-eed5-4033-88ff-eed4fedb2bd4" containerName="maas-api"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276102    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ff75a9-1621-4578-9351-aefbd6931a12" containerName="main"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276108    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ff75a9-1621-4578-9351-aefbd6931a12" containerName="main"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276118    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d6b984-edb2-4544-aa9b-a07efdb2e989" containerName="main"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276123    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d6b984-edb2-4544-aa9b-a07efdb2e989" containerName="main"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276203    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5bd2a6-338e-4621-a267-08bd1a0d433d" containerName="maas-api"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276211    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e027cd-eed5-4033-88ff-eed4fedb2bd4" containerName="maas-api"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276219    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d6b984-edb2-4544-aa9b-a07efdb2e989" containerName="main"
Jul 07 13:57:25.276252 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.276224    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ff75a9-1621-4578-9351-aefbd6931a12" containerName="main"
Jul 07 13:57:25.279213 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.279190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.282130 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.282109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4dq5n\"/\"openshift-service-ca.crt\""
Jul 07 13:57:25.282281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.282235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4dq5n\"/\"kube-root-ca.crt\""
Jul 07 13:57:25.282281 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.282247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4dq5n\"/\"default-dockercfg-5mq4r\""
Jul 07 13:57:25.288993 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.288970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4dq5n/must-gather-cdwbk"]
Jul 07 13:57:25.293621 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.293597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:25.321686 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.321663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rvr\" (UniqueName: \"kubernetes.io/projected/92435dbb-50a9-4dd1-920b-9d7c899df575-kube-api-access-48rvr\") pod \"must-gather-cdwbk\" (UID: \"92435dbb-50a9-4dd1-920b-9d7c899df575\") " pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.321804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.321732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92435dbb-50a9-4dd1-920b-9d7c899df575-must-gather-output\") pod \"must-gather-cdwbk\" (UID: \"92435dbb-50a9-4dd1-920b-9d7c899df575\") " pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.391020 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.390997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log" limit=1024
Jul 07 13:57:25.423001 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.422978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rvr\" (UniqueName: \"kubernetes.io/projected/92435dbb-50a9-4dd1-920b-9d7c899df575-kube-api-access-48rvr\") pod \"must-gather-cdwbk\" (UID: \"92435dbb-50a9-4dd1-920b-9d7c899df575\") " pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.423089 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.423027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92435dbb-50a9-4dd1-920b-9d7c899df575-must-gather-output\") pod \"must-gather-cdwbk\" (UID: \"92435dbb-50a9-4dd1-920b-9d7c899df575\") " pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.423360 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.423345    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92435dbb-50a9-4dd1-920b-9d7c899df575-must-gather-output\") pod \"must-gather-cdwbk\" (UID: \"92435dbb-50a9-4dd1-920b-9d7c899df575\") " pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.430730 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.430706    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rvr\" (UniqueName: \"kubernetes.io/projected/92435dbb-50a9-4dd1-920b-9d7c899df575-kube-api-access-48rvr\") pod \"must-gather-cdwbk\" (UID: \"92435dbb-50a9-4dd1-920b-9d7c899df575\") " pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.589716 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.589647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4dq5n/must-gather-cdwbk"
Jul 07 13:57:25.721307 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.721282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4dq5n/must-gather-cdwbk"]
Jul 07 13:57:25.723554 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:57:25.723523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92435dbb_50a9_4dd1_920b_9d7c899df575.slice/crio-f2ed98b405f35c9304cb3373e3e447de5b91fcb358a6df1ddfdd67cd2a171e2c WatchSource:0}: Error finding container f2ed98b405f35c9304cb3373e3e447de5b91fcb358a6df1ddfdd67cd2a171e2c: Status 404 returned error can't find the container with id f2ed98b405f35c9304cb3373e3e447de5b91fcb358a6df1ddfdd67cd2a171e2c
Jul 07 13:57:25.725722 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:25.725705    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 13:57:26.087235 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:26.087196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4dq5n/must-gather-cdwbk" event={"ID":"92435dbb-50a9-4dd1-920b-9d7c899df575","Type":"ContainerStarted","Data":"f2ed98b405f35c9304cb3373e3e447de5b91fcb358a6df1ddfdd67cd2a171e2c"}
Jul 07 13:57:27.094546 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:27.094507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4dq5n/must-gather-cdwbk" event={"ID":"92435dbb-50a9-4dd1-920b-9d7c899df575","Type":"ContainerStarted","Data":"561420fc3701a9737f45240a1449d99de0bfc7a1624c7359d2ef61b2afb5e1d4"}
Jul 07 13:57:27.094901 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:27.094556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4dq5n/must-gather-cdwbk" event={"ID":"92435dbb-50a9-4dd1-920b-9d7c899df575","Type":"ContainerStarted","Data":"ceaa7d3c65812db90fd6dc6dd7e92cc59b1757cce4af438e6d92976efa25a607"}
Jul 07 13:57:27.110270 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:27.110225    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4dq5n/must-gather-cdwbk" podStartSLOduration=1.52547998 podStartE2EDuration="2.110209545s" totalImagesPullingTime="584.729565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 13:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:57:27.108888664 +0000 UTC m=+2452.989259960" watchObservedRunningTime="2026-07-07 13:57:27.110209545 +0000 UTC m=+2452.990580839"
Jul 07 13:57:27.829720 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:27.829693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mbpsm_81f506f0-9be6-48c3-b771-9e7648ada7a4/global-pull-secret-syncer/0.log"
Jul 07 13:57:27.916625 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:27.916590    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bdq6p_2a209626-ef8e-403b-b8e4-4778a11c9e2d/konnectivity-agent/0.log"
Jul 07 13:57:28.031695 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:28.031658    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-157.ec2.internal_a8cad6aa2145f5ea3bbbbae2bfaaf983/haproxy/0.log"
Jul 07 13:57:33.499607 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:33.499581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-tsdhk_13d458d1-fdf4-4a80-9666-0e5b949c93f3/kuadrant-console-plugin/0.log"
Jul 07 13:57:33.769350 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:33.769224    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dp5l6_1a1140db-844d-4c86-ba2b-47264b20f075/manager/0.log"
Jul 07 13:57:35.726056 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:35.725939    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-krdlm_23094b01-90f4-46b1-9940-05bd7c0fbe16/kube-state-metrics/0.log"
Jul 07 13:57:35.739697 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:35.739661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-krdlm_23094b01-90f4-46b1-9940-05bd7c0fbe16/kube-rbac-proxy-main/0.log"
Jul 07 13:57:35.756638 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:35.756612    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-krdlm_23094b01-90f4-46b1-9940-05bd7c0fbe16/kube-rbac-proxy-self/0.log"
Jul 07 13:57:35.785378 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:35.785349    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64db89fcd9-jdqlw_1a039c05-d8f5-4f2d-9dc4-f1d9c94d6d5b/metrics-server/0.log"
Jul 07 13:57:35.998109 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:35.998075    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nszwn_10691b1d-77e6-414f-b4a4-d204de1be760/node-exporter/0.log"
Jul 07 13:57:36.014766 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.014736    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nszwn_10691b1d-77e6-414f-b4a4-d204de1be760/kube-rbac-proxy/0.log"
Jul 07 13:57:36.034372 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.034344    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nszwn_10691b1d-77e6-414f-b4a4-d204de1be760/init-textfile/0.log"
Jul 07 13:57:36.056309 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.056224    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-g9z2m_efecd0e8-7e87-47d1-aa13-5f7e2815adf0/kube-rbac-proxy-main/0.log"
Jul 07 13:57:36.073675 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.073639    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-g9z2m_efecd0e8-7e87-47d1-aa13-5f7e2815adf0/kube-rbac-proxy-self/0.log"
Jul 07 13:57:36.090920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.090889    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-g9z2m_efecd0e8-7e87-47d1-aa13-5f7e2815adf0/openshift-state-metrics/0.log"
Jul 07 13:57:36.128752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.128704    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/prometheus/0.log"
Jul 07 13:57:36.146711 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.146682    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/config-reloader/0.log"
Jul 07 13:57:36.164693 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.164659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/thanos-sidecar/0.log"
Jul 07 13:57:36.185456 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.185430    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/kube-rbac-proxy-web/0.log"
Jul 07 13:57:36.204111 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.204079    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/kube-rbac-proxy/0.log"
Jul 07 13:57:36.222177 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.222115    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/kube-rbac-proxy-thanos/0.log"
Jul 07 13:57:36.240006 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.239978    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_62596082-a023-4eab-b3ff-161ef7fd16fd/init-config-reloader/0.log"
Jul 07 13:57:36.269465 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.269378    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-r8nms_c7f94f31-2469-448f-8e9c-0180a0de52cc/prometheus-operator/0.log"
Jul 07 13:57:36.284178 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.284128    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-r8nms_c7f94f31-2469-448f-8e9c-0180a0de52cc/kube-rbac-proxy/0.log"
Jul 07 13:57:36.311943 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.311910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-79qth_21ad6ceb-6375-49dc-893b-b7c6e2353305/prometheus-operator-admission-webhook/0.log"
Jul 07 13:57:36.402694 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.402661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f6f8f7f6-tbwx5_259a1bfa-4469-4ad4-8742-35e0ae7f1da8/thanos-query/0.log"
Jul 07 13:57:36.424822 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.424792    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f6f8f7f6-tbwx5_259a1bfa-4469-4ad4-8742-35e0ae7f1da8/kube-rbac-proxy-web/0.log"
Jul 07 13:57:36.443276 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.443247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f6f8f7f6-tbwx5_259a1bfa-4469-4ad4-8742-35e0ae7f1da8/kube-rbac-proxy/0.log"
Jul 07 13:57:36.461826 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.461799    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f6f8f7f6-tbwx5_259a1bfa-4469-4ad4-8742-35e0ae7f1da8/prom-label-proxy/0.log"
Jul 07 13:57:36.479744 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.479706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f6f8f7f6-tbwx5_259a1bfa-4469-4ad4-8742-35e0ae7f1da8/kube-rbac-proxy-rules/0.log"
Jul 07 13:57:36.497646 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:36.497618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84f6f8f7f6-tbwx5_259a1bfa-4469-4ad4-8742-35e0ae7f1da8/kube-rbac-proxy-metrics/0.log"
Jul 07 13:57:38.195938 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.195898    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"]
Jul 07 13:57:38.203763 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.203736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.206518 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.206494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"]
Jul 07 13:57:38.357323 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.357287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-podres\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.357628 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.357611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqz9\" (UniqueName: \"kubernetes.io/projected/b5a1869e-4f66-41d8-b839-d81ce506b29c-kube-api-access-cnqz9\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.357828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.357808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-sys\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.357998 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.357981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-proc\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.358190 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.358142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-lib-modules\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459135 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-sys\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459135 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-proc\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-lib-modules\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-podres\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqz9\" (UniqueName: \"kubernetes.io/projected/b5a1869e-4f66-41d8-b839-d81ce506b29c-kube-api-access-cnqz9\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459266    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-proc\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459322    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-lib-modules\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459398 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459202    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-sys\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.459725 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.459451    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5a1869e-4f66-41d8-b839-d81ce506b29c-podres\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.468651 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.468607    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqz9\" (UniqueName: \"kubernetes.io/projected/b5a1869e-4f66-41d8-b839-d81ce506b29c-kube-api-access-cnqz9\") pod \"perf-node-gather-daemonset-j5cdg\" (UID: \"b5a1869e-4f66-41d8-b839-d81ce506b29c\") " pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.518658 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.518620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:38.718002 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.717693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"]
Jul 07 13:57:38.720786 ip-10-0-133-157 kubenswrapper[2581]: W0707 13:57:38.720747    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb5a1869e_4f66_41d8_b839_d81ce506b29c.slice/crio-d4b9a15c1afacf913bc7d8fb0793751c9c6883b65bd51b14dd6351174e42a339 WatchSource:0}: Error finding container d4b9a15c1afacf913bc7d8fb0793751c9c6883b65bd51b14dd6351174e42a339: Status 404 returned error can't find the container with id d4b9a15c1afacf913bc7d8fb0793751c9c6883b65bd51b14dd6351174e42a339
Jul 07 13:57:38.878642 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:38.878613    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-kztcl_8f8426c2-3e6c-43f1-950e-8ee0cfe3fcb1/download-server/0.log"
Jul 07 13:57:39.183804 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:39.183769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg" event={"ID":"b5a1869e-4f66-41d8-b839-d81ce506b29c","Type":"ContainerStarted","Data":"1df01a26ad90023dcec3edc826753f7137ea84c5215fdee11b8728bb5e173a4d"}
Jul 07 13:57:39.184003 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:39.183824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg" event={"ID":"b5a1869e-4f66-41d8-b839-d81ce506b29c","Type":"ContainerStarted","Data":"d4b9a15c1afacf913bc7d8fb0793751c9c6883b65bd51b14dd6351174e42a339"}
Jul 07 13:57:39.184003 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:39.183909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:39.199931 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:39.199881    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg" podStartSLOduration=1.199866063 podStartE2EDuration="1.199866063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 13:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 13:57:39.197842298 +0000 UTC m=+2465.078213631" watchObservedRunningTime="2026-07-07 13:57:39.199866063 +0000 UTC m=+2465.080237356"
Jul 07 13:57:40.520882 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:40.520849    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hbz7c_7b9e457f-cf36-4628-8e9f-9496d8111b71/dns/0.log"
Jul 07 13:57:40.539231 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:40.539199    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hbz7c_7b9e457f-cf36-4628-8e9f-9496d8111b71/kube-rbac-proxy/0.log"
Jul 07 13:57:40.577927 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:40.577899    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c48gn_a4845995-0d75-4296-ad03-aea66ec32c37/dns-node-resolver/0.log"
Jul 07 13:57:41.120874 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:41.120839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-bddc7bbc5-khlbj_bc338116-d8e0-4bfd-a457-d2b7254994d2/registry/0.log"
Jul 07 13:57:41.140964 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:41.140930    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ksnhj_b20eff8d-cbbe-4869-b380-13a25e5fabd0/node-ca/0.log"
Jul 07 13:57:42.322637 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:42.322609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577c5f8c7-qxgnv_5540609e-e8e9-483f-8ce9-57ff0fc14da7/kube-auth-proxy/0.log"
Jul 07 13:57:42.413752 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:42.413692    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-4nrs6_288a9386-9eb8-4146-9d36-1f1da3098d34/payload-pre-processing/0.log"
Jul 07 13:57:43.173801 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:43.173767    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnwlb_bf83232b-8e51-4f9d-8c9e-b9d5b68e94e2/serve-healthcheck-canary/0.log"
Jul 07 13:57:43.624748 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:43.624721    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2fvnk_4c9f9d2c-f1b8-41aa-af80-7854ce25f309/kube-rbac-proxy/0.log"
Jul 07 13:57:43.640784 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:43.640761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2fvnk_4c9f9d2c-f1b8-41aa-af80-7854ce25f309/exporter/0.log"
Jul 07 13:57:43.661151 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:43.661115    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2fvnk_4c9f9d2c-f1b8-41aa-af80-7854ce25f309/extractor/0.log"
Jul 07 13:57:45.200211 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:45.200153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4dq5n/perf-node-gather-daemonset-j5cdg"
Jul 07 13:57:46.304828 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:46.304796    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fbc88fbc-q5ghb_0522fb53-79eb-47f6-ac7a-2553b6d7c868/manager/0.log"
Jul 07 13:57:46.321702 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:46.321673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69fbc88fbc-q5ghb_0522fb53-79eb-47f6-ac7a-2553b6d7c868/copy-manifests/0.log"
Jul 07 13:57:48.053810 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:48.053776    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76d5cc9db8-jd7bq_c6963abc-7177-4904-ae7b-68867e0eb627/manager/0.log"
Jul 07 13:57:48.872385 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:48.872355    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577c5f8c7-qxgnv_5540609e-e8e9-483f-8ce9-57ff0fc14da7/kube-auth-proxy/0.log"
Jul 07 13:57:48.947113 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:48.947078    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-4nrs6_288a9386-9eb8-4146-9d36-1f1da3098d34/payload-pre-processing/0.log"
Jul 07 13:57:50.416861 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:50.416818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577c5f8c7-qxgnv_5540609e-e8e9-483f-8ce9-57ff0fc14da7/kube-auth-proxy/0.log"
Jul 07 13:57:50.489530 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:50.489490    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-4nrs6_288a9386-9eb8-4146-9d36-1f1da3098d34/payload-pre-processing/0.log"
Jul 07 13:57:54.461860 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.461772    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4srzl_4a5d83cc-f411-41ea-b874-b2f5aeb7426c/kube-multus/0.log"
Jul 07 13:57:54.838678 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.838654    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/kube-multus-additional-cni-plugins/0.log"
Jul 07 13:57:54.865491 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.865462    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/egress-router-binary-copy/0.log"
Jul 07 13:57:54.889010 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.888983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/cni-plugins/0.log"
Jul 07 13:57:54.909002 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.908977    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/bond-cni-plugin/0.log"
Jul 07 13:57:54.933908 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.933878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/routeoverride-cni/0.log"
Jul 07 13:57:54.958883 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.958855    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/whereabouts-cni-bincopy/0.log"
Jul 07 13:57:54.980663 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:54.980634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-z9bz7_7e9315e0-f682-4632-acf8-0574443059e8/whereabouts-cni/0.log"
Jul 07 13:57:55.090723 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:55.090640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s6p2q_404f302b-7cf7-41ff-a9d1-a2ada86d1ab5/network-metrics-daemon/0.log"
Jul 07 13:57:55.105920 ip-10-0-133-157 kubenswrapper[2581]: I0707 13:57:55.105900    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s6p2q_404f302b-7cf7-41ff-a9d1-a2ada86d1ab5/kube-rbac-proxy/0.log"