Nov 04 03:14:24.870754 ip-10-0-132-184 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 04 03:14:24.870765 ip-10-0-132-184 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 04 03:14:24.870774 ip-10-0-132-184 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 04 03:14:24.871108 ip-10-0-132-184 systemd[1]: Failed to start Kubernetes Kubelet. Nov 04 03:14:34.875589 ip-10-0-132-184 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 04 03:14:34.875602 ip-10-0-132-184 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot faa3cc94e0e14d379065e71a40d736ef -- Nov 04 03:16:19.583775 ip-10-0-132-184 systemd[1]: Starting Kubernetes Kubelet... Nov 04 03:16:20.007770 ip-10-0-132-184 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:16:20.007770 ip-10-0-132-184 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 03:16:20.007770 ip-10-0-132-184 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:16:20.007770 ip-10-0-132-184 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:16:20.012224 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012147 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012227 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012233 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012236 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012239 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012241 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012244 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012246 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012254 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012257 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012260 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012262 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012265 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012267 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012270 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.012264 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012273 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012276 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012279 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012281 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012284 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012286 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012289 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012292 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012295 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012297 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012300 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012302 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012305 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012307 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012310 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012312 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012315 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012317 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012319 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.012666 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012322 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012324 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012327 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012329 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012332 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012335 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012337 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012339 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012342 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012344 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012346 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012349 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012351 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012354 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012357 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012360 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012362 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012365 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012367 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012370 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.013131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012372 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012375 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012378 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012381 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012383 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012385 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012388 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012390 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012393 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012395 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012397 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012400 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012402 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012404 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012407 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012409 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012412 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012415 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012418 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012420 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.013616 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012423 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012425 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012428 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012430 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012432 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012437 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012441 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012445 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012448 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012452 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012455 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012459 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012461 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012463 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012482 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012485 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012488 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012491 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012493 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.014182 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012496 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012499 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012501 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012504 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012506 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012509 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012511 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012516 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012519 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012522 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012525 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.012528 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013326 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013333 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013336 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013339 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013342 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013351 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013356 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.014657 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013360 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013364 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013366 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013369 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013372 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013375 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013378 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013380 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013383 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013385 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013388 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013390 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013393 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013395 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013397 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013400 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013402 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013404 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013407 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.015100 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013409 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013412 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013414 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013417 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013419 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013422 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013432 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013435 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013437 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013440 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013442 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013445 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013447 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013450 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013452 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013455 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013458 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013461 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013463 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.015561 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013482 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013485 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013487 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013490 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013492 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013495 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013498 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013500 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013504 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013506 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013509 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013511 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013514 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013516 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013519 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013522 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013524 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013526 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013530 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013533 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.016052 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013541 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013545 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013549 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013552 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013555 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013558 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013561 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013563 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013565 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013568 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013570 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013573 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013576 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013578 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013581 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013583 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013585 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013588 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013590 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.016541 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013593 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013595 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013598 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013600 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013603 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013605 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013608 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013610 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013612 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013616 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013618 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013621 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013624 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013627 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013630 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013632 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013635 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013638 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013640 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.017002 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013642 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013645 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.013647 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014170 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014176 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014179 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014182 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014185 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014188 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014191 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014194 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014196 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014199 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014202 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014204 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014207 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014209 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014212 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014214 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.017444 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014216 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014219 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014221 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014224 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014226 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014230 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014233 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014235 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014239 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014241 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014244 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014246 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014248 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014251 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014253 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014256 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014258 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014261 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014264 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.017903 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014266 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014269 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014271 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014274 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014276 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014278 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014281 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014283 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014286 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014288 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014290 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014293 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014295 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014298 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014300 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014303 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014305 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014308 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014310 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014312 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.018377 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014315 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014318 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014322 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014325 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014328 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014332 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014334 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014336 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014339 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014341 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014345 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014347 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014350 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014352 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014355 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014357 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014359 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014362 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014365 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.018867 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014367 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014369 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014372 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014374 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014376 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014379 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014382 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014384 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014387 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014392 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014395 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014397 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014400 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014402 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014405 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014407 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014411 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014413 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014415 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014418 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.019307 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014420 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014423 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014425 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014429 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014432 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014435 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014438 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014440 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014443 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014445 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014448 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.014450 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015459 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015482 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015490 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015495 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015500 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015503 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015507 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015512 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015515 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 03:16:20.019786 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015519 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015522 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015526 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015529 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015532 2579 flags.go:64] FLAG: --cgroup-root="" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015535 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015538 2579 flags.go:64] FLAG: --client-ca-file="" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015540 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015544 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015548 2579 flags.go:64] FLAG: --cluster-domain="" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015551 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015554 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015557 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015562 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015566 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015569 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015572 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015576 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015579 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015581 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015584 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015588 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015590 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015595 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 03:16:20.020266 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015597 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015600 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015603 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015606 2579 flags.go:64] FLAG: --enable-server="true" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015609 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015614 2579 flags.go:64] FLAG: --event-burst="100" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015617 2579 flags.go:64] FLAG: --event-qps="50" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015620 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015624 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015627 2579 flags.go:64] FLAG: --eviction-hard="" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015632 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015634 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015637 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015640 2579 flags.go:64] FLAG: --eviction-soft="" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015643 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015646 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015649 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015657 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015662 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015665 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015668 2579 flags.go:64] FLAG: --feature-gates="" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015673 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015676 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015679 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015682 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015685 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 04 03:16:20.020890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015688 2579 flags.go:64] FLAG: --help="false" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015691 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015694 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015697 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015700 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015704 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015707 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015710 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015712 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015715 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015718 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015721 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015724 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015727 2579 flags.go:64] FLAG: --kube-reserved="" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015730 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015732 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015735 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015738 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015741 2579 flags.go:64] FLAG: --lock-file="" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015743 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015746 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015749 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015755 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 03:16:20.021518 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015757 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015762 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015765 2579 flags.go:64] FLAG: --logging-format="text" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015768 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015771 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015774 2579 flags.go:64] FLAG: --manifest-url="" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015777 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015781 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015784 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015788 2579 flags.go:64] FLAG: --max-pods="110" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015791 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015794 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015797 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015800 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015803 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015806 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015809 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015816 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015819 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015822 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015825 2579 flags.go:64] FLAG: --pod-cidr="" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015828 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015831 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015834 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015837 2579 flags.go:64] FLAG: --port="10250" Nov 04 03:16:20.022063 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015840 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015842 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0d5a0a98334663306" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015845 2579 flags.go:64] FLAG: --qos-reserved="" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015848 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015851 2579 flags.go:64] FLAG: --register-node="true" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015854 2579 flags.go:64] FLAG: --register-with-taints="" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015857 2579 flags.go:64] FLAG: --registry-burst="10" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015860 2579 flags.go:64] FLAG: --registry-qps="5" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015864 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015867 2579 flags.go:64] FLAG: --reserved-memory="" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015871 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015874 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015877 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015880 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015882 2579 flags.go:64] FLAG: --runonce="false" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015885 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015888 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015891 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015894 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015897 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015900 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015903 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015906 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015909 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015912 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 03:16:20.022717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015914 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015917 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015920 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015923 2579 flags.go:64] FLAG: --system-cgroups="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015927 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015933 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015935 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015938 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015942 2579 flags.go:64] FLAG: --tls-min-version="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015944 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015947 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015950 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015953 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015956 2579 flags.go:64] FLAG: --v="2" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015961 2579 flags.go:64] FLAG: --version="false" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015966 2579 flags.go:64] FLAG: --vmodule="" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015970 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.015973 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016109 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016113 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016116 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016119 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016121 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016124 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.023768 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016126 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016129 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016131 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016134 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016139 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016141 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016144 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016146 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016148 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016151 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016154 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016157 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016160 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016162 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016165 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016167 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016170 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016173 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016175 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.024799 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016178 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016180 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016182 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016187 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016189 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016192 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016195 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016197 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016200 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016202 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016205 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016207 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016210 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016212 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016215 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016217 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016220 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016223 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016225 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.025498 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016228 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016230 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016232 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016235 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016237 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016241 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016243 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016246 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016248 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016251 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016253 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016255 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016259 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016263 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016266 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016270 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016275 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016278 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016281 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.026232 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016283 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016287 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016289 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016292 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016294 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016296 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016299 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016302 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016304 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016306 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016309 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016311 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016314 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016316 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016319 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016321 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016324 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016326 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016330 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016332 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.027004 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016335 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016338 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016340 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016343 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016346 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016348 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016350 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016353 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016355 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016359 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016362 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016364 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016367 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016370 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016373 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016375 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016378 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016381 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016383 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016386 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.027876 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016388 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016391 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.016416 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016453 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016457 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016461 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016463 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016479 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016481 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016484 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016488 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016491 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016493 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016496 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016499 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016502 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.028766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016504 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016506 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016509 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016512 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016514 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016517 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016521 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016524 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016526 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016529 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016531 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016534 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016537 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016539 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016542 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016545 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016547 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016550 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016553 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016555 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.029270 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016558 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016560 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016563 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016565 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016568 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016570 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016574 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016576 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016579 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016581 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016584 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016587 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016589 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016592 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016594 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016596 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016599 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016601 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016605 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.029815 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016607 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016610 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016612 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016614 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016617 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016620 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016622 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016625 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016627 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016630 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016632 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016635 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016637 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016640 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016642 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016645 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016647 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016649 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016652 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016655 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.030804 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016658 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016660 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016663 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016665 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016668 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016672 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016675 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016679 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016687 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016690 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016693 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016698 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016701 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016704 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016706 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016709 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016711 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016714 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016717 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.031490 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016720 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016722 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016725 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016727 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016729 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016732 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016734 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016737 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016740 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016742 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016745 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016747 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016751 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016753 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.016757 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.026864 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.026885 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.026915 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.026923 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 03:16:20.032017 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027010 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027018 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027024 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027028 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027032 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027036 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027041 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027045 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027049 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027054 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027058 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027063 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027067 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027074 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027080 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027086 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027094 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027098 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027103 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.032570 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027108 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027112 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027116 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027120 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027125 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027129 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027133 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027137 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027142 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027146 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027150 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027155 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027159 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027165 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027169 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027173 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027177 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027182 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027186 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.033043 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027190 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027194 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027198 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027202 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027207 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027212 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027216 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027220 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027225 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027230 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027238 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027242 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027246 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027250 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027255 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027259 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027263 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027267 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027271 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.033575 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027275 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027280 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027284 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027288 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027292 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027297 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027301 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027306 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027310 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027314 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027318 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027322 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027326 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027330 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027334 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027339 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027343 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027347 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027351 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027355 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027359 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.034131 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027365 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027369 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027374 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027378 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027382 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027387 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027391 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027395 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027399 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027404 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027408 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027413 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027417 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027422 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027426 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027430 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027435 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027439 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027445 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.034645 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027451 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027456 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027461 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027483 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027488 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027493 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027497 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027501 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.027544 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027777 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027787 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027792 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027797 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027803 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027808 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027813 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.035328 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027817 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027822 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027826 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027830 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027834 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027839 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027843 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027847 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027851 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027855 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027859 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027864 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027868 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027872 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027876 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027879 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027884 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027888 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027892 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027896 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.035984 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027900 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027904 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027908 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027912 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027916 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027922 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027926 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027930 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027935 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027939 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027943 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027948 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027953 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027958 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027962 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027966 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027970 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027974 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027978 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027982 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.036698 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027986 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027991 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.027995 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028000 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028004 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028008 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028012 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028019 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028026 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028031 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028036 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028041 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028045 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028050 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028054 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028058 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028062 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028066 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028070 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.037157 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028074 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028079 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028083 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028087 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028091 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028097 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028101 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028106 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028110 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028114 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028119 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028123 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028127 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028131 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028135 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028139 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028143 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028147 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028151 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.037766 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028155 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028160 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028164 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028169 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028173 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028177 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028184 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028189 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028194 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028198 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028203 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028207 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028211 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028215 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028219 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028224 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028229 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028233 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028237 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.038242 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028241 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028246 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.028285 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028335 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028343 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028347 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028352 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028356 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028360 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028364 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028368 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028372 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028376 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028380 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028384 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028388 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:16:20.038705 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028392 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028397 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028401 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028406 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028410 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028415 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028419 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028423 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028427 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028431 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028435 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028439 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028443 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028447 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028451 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028456 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028460 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028483 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028489 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028493 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:16:20.039136 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028497 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028501 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028505 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028509 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028513 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028517 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028521 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028525 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028529 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028533 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028537 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028541 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028548 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028555 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028560 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028564 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028568 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028573 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028577 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:16:20.039944 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028581 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028585 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028589 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028593 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028597 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028601 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028605 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028609 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028613 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028617 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028622 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028632 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028636 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028640 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028644 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028649 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028653 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028657 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028661 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028665 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:16:20.040388 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028669 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028673 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028677 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028681 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028685 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028690 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028694 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028699 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028703 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028707 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028714 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028721 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028725 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028729 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028733 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028738 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028742 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028746 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028750 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:16:20.040886 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028755 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028759 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028763 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028767 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028771 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028783 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028787 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028790 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028794 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028799 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028805 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028809 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028816 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028820 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:20.028825 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.028921 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.031957 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.033036 2579 server.go:1014] "Starting client certificate rotation" Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.033130 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:16:20.041350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.033970 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:16:20.041801 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.035532 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:16:20.041801 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.040837 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:16:20.055061 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.055044 2579 log.go:25] "Validated CRI v1 runtime API" Nov 04 03:16:20.058696 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.058680 2579 log.go:25] "Validated CRI v1 image API" Nov 04 03:16:20.059551 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.059523 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 03:16:20.064280 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.064258 2579 fs.go:135] Filesystem UUIDs: map[71cacd08-7b0a-4ba2-8e1e-d67a070fd986:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89e34c74-3900-46a4-acb2-f8bed21ec0ac:/dev/nvme0n1p3] Nov 04 03:16:20.064364 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.064280 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 04 03:16:20.069079 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.068958 2579 manager.go:217] Machine: {Timestamp:2026-11-04 03:16:20.068456162 +0000 UTC m=+0.361325887 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100912 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20934aeeddb2377520963ad2beab78 SystemUUID:ec20934a-eedd-b237-7520-963ad2beab78 BootID:faa3cc94-e0e1-4d37-9065-e71a40d736ef Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:36:4c:b0:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:36:4c:b0:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:b2:fd:12:5e:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 03:16:20.069689 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.069676 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 03:16:20.069827 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.069812 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 03:16:20.072371 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.072345 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 03:16:20.072557 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.072373 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-184.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 03:16:20.072638 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.072561 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 03:16:20.072638 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.072574 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 03:16:20.072638 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.072596 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:16:20.073566 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.073554 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:16:20.074779 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.074767 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 03:16:20.079326 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.079312 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 03:16:20.079464 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.079453 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 03:16:20.080263 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.080253 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 04 03:16:20.080323 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.080272 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 03:16:20.080323 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.080285 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 03:16:20.080323 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.080299 2579 kubelet.go:405] "Adding apiserver pod source" Nov 04 03:16:20.080323 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.080310 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 03:16:20.080677 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.080663 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:16:20.083920 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.083772 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 03:16:20.091085 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.091071 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:16:20.091861 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.091849 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 03:16:20.091898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.091880 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 03:16:20.092035 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092025 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 03:16:20.092075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092041 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 03:16:20.092075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092047 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 03:16:20.092075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092052 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 03:16:20.092075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092058 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 03:16:20.092075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092064 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 03:16:20.092711 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092701 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 03:16:20.092746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092711 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 03:16:20.092746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092719 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 03:16:20.092746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092725 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 03:16:20.092746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092734 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 03:16:20.092746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.092742 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 03:16:20.093621 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.093612 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 03:16:20.093656 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.093621 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 03:16:20.097335 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.097322 2579 server.go:1263] "Started kubelet" Nov 04 03:16:20.097443 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.097396 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 03:16:20.097509 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.097435 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 03:16:20.097551 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.097513 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 03:16:20.098132 ip-10-0-132-184 systemd[1]: Started Kubernetes Kubelet. Nov 04 03:16:20.099257 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.099162 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 03:16:20.100386 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.100370 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 04 03:16:20.105047 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.105025 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 04 03:16:20.106428 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.106412 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 03:16:20.107699 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.107685 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 03:16:20.107841 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.107807 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:20.108511 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108496 2579 factory.go:55] Registering systemd factory Nov 04 03:16:20.108511 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108512 2579 factory.go:223] Registration of the systemd container factory successfully Nov 04 03:16:20.108676 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108663 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 03:16:20.108731 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108678 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 03:16:20.108731 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108700 2579 factory.go:153] Registering CRI-O factory Nov 04 03:16:20.108731 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108715 2579 factory.go:223] Registration of the crio container factory successfully Nov 04 03:16:20.108731 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108660 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 03:16:20.108905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108769 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 03:16:20.108905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108783 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 04 03:16:20.108905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108788 2579 factory.go:103] Registering Raw factory Nov 04 03:16:20.108905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108802 2579 manager.go:1199] Started watching for new ooms in manager Nov 04 03:16:20.108905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.108789 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 04 03:16:20.109872 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.109858 2579 manager.go:319] Starting recovery of all containers Nov 04 03:16:20.120569 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.120551 2579 manager.go:324] Recovery completed Nov 04 03:16:20.125250 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.125237 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.126668 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.126647 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.126722 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.126682 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.126722 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.126693 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.127161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127147 2579 cpu_manager.go:226] "Starting" policy="none" Nov 04 03:16:20.127161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127157 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 03:16:20.127258 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127172 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 03:16:20.127258 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127201 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.127920 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127907 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.127974 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127931 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.127974 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.127941 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.129305 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.129293 2579 policy_none.go:50] "Start" Nov 04 03:16:20.129361 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.129309 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 03:16:20.129361 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.129318 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 03:16:20.131247 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.131236 2579 policy_none.go:44] "Start" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.174897 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.183255 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.184556 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.184575 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.184593 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.184623 2579 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.189854 2579 manager.go:341] "Starting Device Plugin manager" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.189880 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 03:16:20.190048 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.189888 2579 server.go:87] "Starting device plugin registration server" Nov 04 03:16:20.190515 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.190101 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 03:16:20.190515 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.190110 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 03:16:20.190515 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.190231 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 03:16:20.190515 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.190310 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 03:16:20.190515 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.190318 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 03:16:20.190732 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.190713 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 03:16:20.190787 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.190766 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:20.244044 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.243985 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-184.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 04 03:16:20.244044 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.243985 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-184.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 04 03:16:20.245100 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.243966 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-184.us-west-1.compute.internal.18e4b932f43ab954 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-184.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-184.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 03:16:20.097292628 +0000 UTC m=+0.390162354,LastTimestamp:2026-11-04 03:16:20.097292628 +0000 UTC m=+0.390162354,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-184.us-west-1.compute.internal,}" Nov 04 03:16:20.284953 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.284894 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal"] Nov 04 03:16:20.285031 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.284958 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.285994 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.285977 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.286056 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.286007 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.286056 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.286017 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.288101 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288090 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.288263 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.288306 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288290 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.288811 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288785 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.288905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288812 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.288905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288840 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.288905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288858 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.288905 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288888 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.289087 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.288909 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.290716 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.290700 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.290820 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.290804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.290868 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.290850 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.291375 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291359 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.291454 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291383 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.291454 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291406 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.291454 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291440 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.291578 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291481 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.291578 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291498 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.291578 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.291516 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.324958 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.324941 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.326375 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.326351 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 03:16:20.326450 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.326403 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w89hf" Nov 04 03:16:20.327583 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.327567 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.345586 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.345566 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 04 03:16:20.345586 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.345568 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-184.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 04 03:16:20.345713 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.345696 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 04 03:16:20.345940 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.345872 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-184.us-west-1.compute.internal.18e4b932f5faf4c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-184.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-184.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-184.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 03:16:20.126667971 +0000 UTC m=+0.419537696,LastTimestamp:2026-11-04 03:16:20.126667971 +0000 UTC m=+0.419537696,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-184.us-west-1.compute.internal,}" Nov 04 03:16:20.395230 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.395202 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 04 03:16:20.407303 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.407284 2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-184.us-west-1.compute.internal\" not found" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.409498 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.409482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.409543 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.409507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.409543 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.409523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cf4b9ea98890a60ae37fd39b8c09d27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"7cf4b9ea98890a60ae37fd39b8c09d27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.510188 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.510167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.510257 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.510190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.510257 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.510205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cf4b9ea98890a60ae37fd39b8c09d27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"7cf4b9ea98890a60ae37fd39b8c09d27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.510257 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.510229 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cf4b9ea98890a60ae37fd39b8c09d27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"7cf4b9ea98890a60ae37fd39b8c09d27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.510367 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.510258 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.510367 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.510266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.550663 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.550621 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-184.us-west-1.compute.internal\" not found" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.608323 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.608303 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:20.609237 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.609222 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:20.609289 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.609249 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:20.609289 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.609261 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:20.609289 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.609293 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.629416 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.629394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.632551 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.632536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.668701 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:20.668677 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:20.668792 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.668725 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-184.us-west-1.compute.internal\": node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:20.785849 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.785821 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:20.885946 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.885923 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:20.986337 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:20.986314 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.032856 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.032836 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 04 03:16:21.033240 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.032953 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a0eed642b5924453b9c4f38476315c56-78f2aa2176e6b7a4.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-184.us-west-1.compute.internal.18e4b932f5faf4c3\": read tcp 10.0.132.184:49508->3.18.82.172:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-184.us-west-1.compute.internal.18e4b932f5faf4c3 default 4774 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-184.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-184.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-184.us-west-1.compute.internal,},FirstTimestamp:2026-11-04 03:16:20 +0000 UTC,LastTimestamp:2026-11-04 03:16:20.288801055 +0000 UTC m=+0.581670786,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-184.us-west-1.compute.internal,}" Nov 04 03:16:21.087267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.087250 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.187942 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.187896 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.284767 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.284749 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:16:21.285540 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.285525 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:16:21.285664 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.285554 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:16:21.285664 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.285564 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:16:21.288113 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.288100 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.332047 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.332017 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 03:11:20 +0000 UTC" deadline="2028-08-04 06:27:37.274585256 +0000 UTC" Nov 04 03:16:21.332047 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.332044 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15339h11m15.942545881s" Nov 04 03:16:21.388484 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.388446 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.486350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.486285 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:21.489399 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.489383 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.590105 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:21.590089 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:16:21.601865 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.601852 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:21.603805 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.603791 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:21.608842 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.608827 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:21.667409 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.667387 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:16:21.668356 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.668328 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:16:21.721877 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.721858 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:16:21.825858 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:21.825832 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf4b9ea98890a60ae37fd39b8c09d27.slice/crio-2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26 WatchSource:0}: Error finding container 2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26: Status 404 returned error can't find the container with id 2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26 Nov 04 03:16:21.826184 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:21.826164 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee48ef9881478a9b0ef921b39f5cf13c.slice/crio-57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5 WatchSource:0}: Error finding container 57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5: Status 404 returned error can't find the container with id 57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5 Nov 04 03:16:21.830168 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.830155 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:16:21.835413 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:21.835401 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:16:22.081967 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.081947 2579 apiserver.go:52] "Watching apiserver" Nov 04 03:16:22.107974 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.107945 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 04 03:16:22.178681 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.178654 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 03:16:22.178947 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.178929 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc","openshift-multus/network-metrics-daemon-m9tdv","openshift-network-operator/iptables-alerter-khw7x","openshift-ovn-kubernetes/ovnkube-node-54nv5","kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-8zxlg","openshift-network-diagnostics/network-check-target-gxvcp","openshift-image-registry/node-ca-s9jbx","openshift-multus/multus-svnhr","kube-system/global-pull-secret-syncer-cjhm7","kube-system/konnectivity-agent-5xngn","openshift-dns/node-resolver-7ztjl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-j2p7f"] Nov 04 03:16:22.182393 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.182376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:22.182491 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.182459 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:22.184283 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.184270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.186196 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.186176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.188176 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.188159 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.190141 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.190124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.192213 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.192196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.194264 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.194244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.197336 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.196574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.199553 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.199536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.199623 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.199599 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:22.201625 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.201609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.201695 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.201678 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:22.203703 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.203685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.205767 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.205752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.207735 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.207697 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"ee48ef9881478a9b0ef921b39f5cf13c","Type":"ContainerStarted","Data":"57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5"} Nov 04 03:16:22.207798 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.207745 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"7cf4b9ea98890a60ae37fd39b8c09d27","Type":"ContainerStarted","Data":"2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26"} Nov 04 03:16:22.209536 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.209447 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 03:16:22.219237 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd27k\" (UniqueName: \"kubernetes.io/projected/c66837ee-2658-4ef6-bee5-d15bbe58ebde-kube-api-access-sd27k\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.219298 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.219298 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-kubelet\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.219298 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-conf-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.219433 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.219433 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-modprobe-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.219433 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-socket-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.219433 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-system-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.219433 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lns92\" (UniqueName: \"kubernetes.io/projected/d4132467-8852-45ae-8c94-bf0d78847bc6-kube-api-access-lns92\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e04fdd96-e8ac-4338-933c-1b9dadb56b89-host-slash\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-kubelet-config\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9pz\" (UniqueName: \"kubernetes.io/projected/e04fdd96-e8ac-4338-933c-1b9dadb56b89-kube-api-access-kq9pz\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-dbus\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-conf\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-slash\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-var-lib-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-systemd\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.219717 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-registration-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-cnibin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4132467-8852-45ae-8c94-bf0d78847bc6-hosts-file\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b5da75-9530-4e0b-849d-fa50fdec3241-host\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-tmp\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-multus\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-multus-certs\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcsqq\" (UniqueName: \"kubernetes.io/projected/e4b5da75-9530-4e0b-849d-fa50fdec3241-kube-api-access-kcsqq\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-cni-binary-copy\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.219986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-socket-dir-parent\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-ovn\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovn-node-metrics-cert\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220064 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-sys\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-lib-modules\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-netns\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dab481c8-cff5-4c29-aa52-552434b043bf-agent-certs\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4b5da75-9530-4e0b-849d-fa50fdec3241-serviceca\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-etc-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-kubernetes\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-tuned\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4132467-8852-45ae-8c94-bf0d78847bc6-tmp-dir\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e04fdd96-e8ac-4338-933c-1b9dadb56b89-iptables-alerter-script\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-log-socket\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-script-lib\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8hb\" (UniqueName: \"kubernetes.io/projected/b24d1578-2e1c-4602-a90e-1d86e1b10645-kube-api-access-gx8hb\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.220514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-daemon-config\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-etc-kubernetes\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-node-log\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-bin\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-bin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dab481c8-cff5-4c29-aa52-552434b043bf-konnectivity-ca\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-kubelet\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-var-lib-kubelet\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-host\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-device-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-sys-fs\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2x88\" (UniqueName: \"kubernetes.io/projected/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kube-api-access-s2x88\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-systemd-units\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-config\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-run\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzppv\" (UniqueName: \"kubernetes.io/projected/074a2389-db4e-4779-979e-0b9ba07a65bf-kube-api-access-lzppv\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-k8s-cni-cncf-io\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-hostroot\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2r9\" (UniqueName: \"kubernetes.io/projected/9380c359-acb2-43f6-af23-89baa5bee3b4-kube-api-access-fb2r9\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.220986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-system-cni-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mwd\" (UniqueName: \"kubernetes.io/projected/284dd9ee-7ea1-4c8d-9674-57c15be65836-kube-api-access-c7mwd\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-systemd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-netd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysconfig\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-cnibin\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.221617 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-netns\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.222016 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.221290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-env-overrides\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.232642 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.232624 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztc7x\"" Nov 04 03:16:22.232715 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.232643 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.232715 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.232644 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.232715 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.232654 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 03:16:22.234036 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.234022 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.234129 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.234119 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 03:16:22.264354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264331 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 03:16:22.264448 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264417 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 03:16:22.264514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264447 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.264619 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264608 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.264688 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264676 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 03:16:22.264726 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264703 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 03:16:22.264805 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264790 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s72nt\"" Nov 04 03:16:22.264836 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264804 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7jn7l\"" Nov 04 03:16:22.264902 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264890 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.264951 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264890 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.264951 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264912 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.264951 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264916 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5q5qr\"" Nov 04 03:16:22.265028 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.264979 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 03:16:22.273498 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.273458 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8nk47\"" Nov 04 03:16:22.273582 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.273508 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 03:16:22.273582 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.273530 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 03:16:22.273582 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.273530 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.273992 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.273977 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj8mj\"" Nov 04 03:16:22.274842 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.274828 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.274888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.274830 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 03:16:22.274888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.274859 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 03:16:22.278490 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278455 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qhl58\"" Nov 04 03:16:22.278490 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278483 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c548\"" Nov 04 03:16:22.278607 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278510 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 03:16:22.278607 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278539 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.278671 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278612 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 03:16:22.278671 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278628 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.278671 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278646 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5pzz6\"" Nov 04 03:16:22.278758 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278683 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 03:16:22.278758 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278700 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 04 03:16:22.278758 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278705 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 03:16:22.278758 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.278733 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k99m7" Nov 04 03:16:22.322037 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.321973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.322164 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4132467-8852-45ae-8c94-bf0d78847bc6-tmp-dir\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.322164 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.322087 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:22.322164 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.322173 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:16:22.822152638 +0000 UTC m=+3.115022370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:22.322354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e04fdd96-e8ac-4338-933c-1b9dadb56b89-iptables-alerter-script\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.322354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-log-socket\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.322354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.322354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-script-lib\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.322354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8hb\" (UniqueName: \"kubernetes.io/projected/b24d1578-2e1c-4602-a90e-1d86e1b10645-kube-api-access-gx8hb\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4132467-8852-45ae-8c94-bf0d78847bc6-tmp-dir\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-log-socket\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-daemon-config\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-etc-kubernetes\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-node-log\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.322637 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-bin\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-bin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dab481c8-cff5-4c29-aa52-552434b043bf-konnectivity-ca\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-kubelet\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-var-lib-kubelet\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-host\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-device-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-sys-fs\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2x88\" (UniqueName: \"kubernetes.io/projected/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kube-api-access-s2x88\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-systemd-units\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-config\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-run\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzppv\" (UniqueName: \"kubernetes.io/projected/074a2389-db4e-4779-979e-0b9ba07a65bf-kube-api-access-lzppv\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.322992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-k8s-cni-cncf-io\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-hostroot\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323033 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-script-lib\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.323220 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2r9\" (UniqueName: \"kubernetes.io/projected/9380c359-acb2-43f6-af23-89baa5bee3b4-kube-api-access-fb2r9\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.323167 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.323216 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret podName:3d6a461d-3472-4285-b5a9-1a11367f6497 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:22.823196412 +0000 UTC m=+3.116066137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret") pod "global-pull-secret-syncer-cjhm7" (UID: "3d6a461d-3472-4285-b5a9-1a11367f6497") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323413 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-system-cni-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mwd\" (UniqueName: \"kubernetes.io/projected/284dd9ee-7ea1-4c8d-9674-57c15be65836-kube-api-access-c7mwd\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323707 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-system-cni-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-daemon-config\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323776 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-device-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-etc-kubernetes\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323915 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-sys-fs\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-systemd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-systemd-units\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.325898 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.323995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-netd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-node-log\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysconfig\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-cnibin\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324067 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-bin\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-bin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-netns\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-env-overrides\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324183 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-netns\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd27k\" (UniqueName: \"kubernetes.io/projected/c66837ee-2658-4ef6-bee5-d15bbe58ebde-kube-api-access-sd27k\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-systemd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-kubelet\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-conf-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-modprobe-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.326502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-socket-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-system-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324409 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-k8s-cni-cncf-io\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-config\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lns92\" (UniqueName: \"kubernetes.io/projected/d4132467-8852-45ae-8c94-bf0d78847bc6-kube-api-access-lns92\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324501 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-run\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e04fdd96-e8ac-4338-933c-1b9dadb56b89-host-slash\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-kubelet-config\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9pz\" (UniqueName: \"kubernetes.io/projected/e04fdd96-e8ac-4338-933c-1b9dadb56b89-kube-api-access-kq9pz\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-dbus\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-conf\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324681 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-kubelet\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-slash\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-kubelet\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324781 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-dbus\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.327161 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324816 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-modprobe-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-var-lib-kubelet\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324853 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-slash\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324883 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-system-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324712 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-hostroot\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.324958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-var-lib-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325010 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e04fdd96-e8ac-4338-933c-1b9dadb56b89-host-slash\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-var-lib-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-host\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-cnibin\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325084 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-kubelet-config\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-socket-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysconfig\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325151 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-netd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325189 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e04fdd96-e8ac-4338-933c-1b9dadb56b89-iptables-alerter-script\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.328017 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-conf\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dab481c8-cff5-4c29-aa52-552434b043bf-konnectivity-ca\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-systemd\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-conf-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-registration-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-cnibin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-registration-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325413 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-cnibin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325444 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-systemd\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4132467-8852-45ae-8c94-bf0d78847bc6-hosts-file\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325444 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4132467-8852-45ae-8c94-bf0d78847bc6-hosts-file\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b5da75-9530-4e0b-849d-fa50fdec3241-host\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b5da75-9530-4e0b-849d-fa50fdec3241-host\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-tmp\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-env-overrides\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.328781 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-multus\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325645 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-multus\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-multus-certs\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.325713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsqq\" (UniqueName: \"kubernetes.io/projected/e4b5da75-9530-4e0b-849d-fa50fdec3241-kube-api-access-kcsqq\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326087 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-multus-certs\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-cni-binary-copy\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-socket-dir-parent\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-ovn\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovn-node-metrics-cert\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326218 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-ovn\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326232 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-socket-dir-parent\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326237 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-sys\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-lib-modules\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-netns\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dab481c8-cff5-4c29-aa52-552434b043bf-agent-certs\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4b5da75-9530-4e0b-849d-fa50fdec3241-serviceca\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.329430 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-etc-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326433 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-kubernetes\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-tuned\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-cni-binary-copy\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326501 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-netns\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326604 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-etc-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326625 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-lib-modules\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326653 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-sys\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326667 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-kubernetes\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.326875 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4b5da75-9530-4e0b-849d-fa50fdec3241-serviceca\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.329383 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-tuned\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.329449 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-tmp\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.329555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovn-node-metrics-cert\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.329888 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.329631 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dab481c8-cff5-4c29-aa52-552434b043bf-agent-certs\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.377353 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.377303 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:22.377353 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.377325 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:22.377353 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.377338 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vl7nn for pod openshift-network-diagnostics/network-check-target-gxvcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:22.377495 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.377396 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn podName:969e817f-6f19-4c15-835c-abcb599d5554 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:22.87737967 +0000 UTC m=+3.170249396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vl7nn" (UniqueName: "kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn") pod "network-check-target-gxvcp" (UID: "969e817f-6f19-4c15-835c-abcb599d5554") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:22.379894 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.379767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mwd\" (UniqueName: \"kubernetes.io/projected/284dd9ee-7ea1-4c8d-9674-57c15be65836-kube-api-access-c7mwd\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.379894 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.379849 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8hb\" (UniqueName: \"kubernetes.io/projected/b24d1578-2e1c-4602-a90e-1d86e1b10645-kube-api-access-gx8hb\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.380350 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.380327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzppv\" (UniqueName: \"kubernetes.io/projected/074a2389-db4e-4779-979e-0b9ba07a65bf-kube-api-access-lzppv\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.380533 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.380494 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lns92\" (UniqueName: \"kubernetes.io/projected/d4132467-8852-45ae-8c94-bf0d78847bc6-kube-api-access-lns92\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.380968 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.380944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2x88\" (UniqueName: \"kubernetes.io/projected/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kube-api-access-s2x88\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.381044 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.381015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd27k\" (UniqueName: \"kubernetes.io/projected/c66837ee-2658-4ef6-bee5-d15bbe58ebde-kube-api-access-sd27k\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.381267 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.381252 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9pz\" (UniqueName: \"kubernetes.io/projected/e04fdd96-e8ac-4338-933c-1b9dadb56b89-kube-api-access-kq9pz\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.381709 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.381696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2r9\" (UniqueName: \"kubernetes.io/projected/9380c359-acb2-43f6-af23-89baa5bee3b4-kube-api-access-fb2r9\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.381758 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.381696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsqq\" (UniqueName: \"kubernetes.io/projected/e4b5da75-9530-4e0b-849d-fa50fdec3241-kube-api-access-kcsqq\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.495498 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.495479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:16:22.503250 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.503235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:16:22.505296 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.505271 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b5da75_9530_4e0b_849d_fa50fdec3241.slice/crio-a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb WatchSource:0}: Error finding container a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb: Status 404 returned error can't find the container with id a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb Nov 04 03:16:22.510539 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.510519 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04fdd96_e8ac_4338_933c_1b9dadb56b89.slice/crio-09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31 WatchSource:0}: Error finding container 09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31: Status 404 returned error can't find the container with id 09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31 Nov 04 03:16:22.524396 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.524378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:16:22.525484 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.525454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:16:22.528680 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.528665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-svnhr" Nov 04 03:16:22.533096 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.533083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:22.534098 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.534079 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24d1578_2e1c_4602_a90e_1d86e1b10645.slice/crio-a70d94156419643d4a9b4536dcf82d36909bbb9b90389740d3d41aa1150452f1 WatchSource:0}: Error finding container a70d94156419643d4a9b4536dcf82d36909bbb9b90389740d3d41aa1150452f1: Status 404 returned error can't find the container with id a70d94156419643d4a9b4536dcf82d36909bbb9b90389740d3d41aa1150452f1 Nov 04 03:16:22.537508 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.537487 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9891bc_c81c_4ce5_afc9_75c07b82176a.slice/crio-d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452 WatchSource:0}: Error finding container d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452: Status 404 returned error can't find the container with id d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452 Nov 04 03:16:22.539540 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.539525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:22.540729 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.540709 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9380c359_acb2_43f6_af23_89baa5bee3b4.slice/crio-f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9 WatchSource:0}: Error finding container f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9: Status 404 returned error can't find the container with id f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9 Nov 04 03:16:22.541798 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.541758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:16:22.542753 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.542738 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66837ee_2658_4ef6_bee5_d15bbe58ebde.slice/crio-d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd WatchSource:0}: Error finding container d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd: Status 404 returned error can't find the container with id d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd Nov 04 03:16:22.547436 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.547418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:16:22.548438 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.548420 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab481c8_cff5_4c29_aa52_552434b043bf.slice/crio-ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f WatchSource:0}: Error finding container ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f: Status 404 returned error can't find the container with id ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f Nov 04 03:16:22.550237 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.550206 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4132467_8852_45ae_8c94_bf0d78847bc6.slice/crio-25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d WatchSource:0}: Error finding container 25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d: Status 404 returned error can't find the container with id 25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d Nov 04 03:16:22.555805 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:22.555790 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284dd9ee_7ea1_4c8d_9674_57c15be65836.slice/crio-fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158 WatchSource:0}: Error finding container fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158: Status 404 returned error can't find the container with id fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158 Nov 04 03:16:22.830825 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.830794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:22.830977 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.830844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:22.830977 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.830934 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:22.830977 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.830934 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:22.831153 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.830987 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret podName:3d6a461d-3472-4285-b5a9-1a11367f6497 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:23.830973011 +0000 UTC m=+4.123842727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret") pod "global-pull-secret-syncer-cjhm7" (UID: "3d6a461d-3472-4285-b5a9-1a11367f6497") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:22.831153 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.831001 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:16:23.830995698 +0000 UTC m=+4.123865409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:22.931714 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:22.931690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:22.931871 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.931837 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:22.931871 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.931857 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:22.931871 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.931866 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vl7nn for pod openshift-network-diagnostics/network-check-target-gxvcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:22.932026 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:22.931916 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn podName:969e817f-6f19-4c15-835c-abcb599d5554 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:23.931899618 +0000 UTC m=+4.224769342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl7nn" (UniqueName: "kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn") pod "network-check-target-gxvcp" (UID: "969e817f-6f19-4c15-835c-abcb599d5554") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:23.193957 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.193920 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xngn" event={"ID":"dab481c8-cff5-4c29-aa52-552434b043bf","Type":"ContainerStarted","Data":"ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f"} Nov 04 03:16:23.195393 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.195356 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd"} Nov 04 03:16:23.196587 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.196561 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9jbx" event={"ID":"e4b5da75-9530-4e0b-849d-fa50fdec3241","Type":"ContainerStarted","Data":"a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb"} Nov 04 03:16:23.197732 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.197708 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerStarted","Data":"fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158"} Nov 04 03:16:23.198756 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.198729 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7ztjl" event={"ID":"d4132467-8852-45ae-8c94-bf0d78847bc6","Type":"ContainerStarted","Data":"25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d"} Nov 04 03:16:23.199869 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.199845 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-svnhr" event={"ID":"9380c359-acb2-43f6-af23-89baa5bee3b4","Type":"ContainerStarted","Data":"f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9"} Nov 04 03:16:23.200975 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.200958 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" event={"ID":"3e9891bc-c81c-4ce5-afc9-75c07b82176a","Type":"ContainerStarted","Data":"d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452"} Nov 04 03:16:23.202088 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.202057 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" event={"ID":"b24d1578-2e1c-4602-a90e-1d86e1b10645","Type":"ContainerStarted","Data":"a70d94156419643d4a9b4536dcf82d36909bbb9b90389740d3d41aa1150452f1"} Nov 04 03:16:23.203180 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.203157 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khw7x" event={"ID":"e04fdd96-e8ac-4338-933c-1b9dadb56b89","Type":"ContainerStarted","Data":"09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31"} Nov 04 03:16:23.281078 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.281019 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:11:22 +0000 UTC" deadline="2028-04-26 17:30:12.06849338 +0000 UTC" Nov 04 03:16:23.281078 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.281075 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12950h13m48.787421484s" Nov 04 03:16:23.837926 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.837890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:23.838090 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.837980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:23.838156 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.838120 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:23.838204 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.838181 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:16:25.838163664 +0000 UTC m=+6.131033397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:23.838274 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.838245 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:23.838330 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.838280 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret podName:3d6a461d-3472-4285-b5a9-1a11367f6497 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:25.838267087 +0000 UTC m=+6.131136814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret") pod "global-pull-secret-syncer-cjhm7" (UID: "3d6a461d-3472-4285-b5a9-1a11367f6497") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:23.938534 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:23.938497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:23.938744 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.938725 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:23.938799 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.938755 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:23.938799 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.938768 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vl7nn for pod openshift-network-diagnostics/network-check-target-gxvcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:23.938915 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:23.938825 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn podName:969e817f-6f19-4c15-835c-abcb599d5554 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:25.938807559 +0000 UTC m=+6.231677287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl7nn" (UniqueName: "kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn") pod "network-check-target-gxvcp" (UID: "969e817f-6f19-4c15-835c-abcb599d5554") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:24.185894 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:24.185862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:24.186126 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:24.185989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:24.186126 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:24.186075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:24.186256 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:24.186154 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:24.186256 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:24.186237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:24.186349 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:24.186313 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:24.282074 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:24.282036 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:11:22 +0000 UTC" deadline="2028-07-17 19:07:52.27934804 +0000 UTC" Nov 04 03:16:24.282074 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:24.282071 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14919h51m27.997281254s" Nov 04 03:16:25.858530 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:25.855750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:25.858530 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:25.856019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:25.858530 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.856137 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:25.858530 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.856194 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:16:29.856176747 +0000 UTC m=+10.149046484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:25.858530 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.856259 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:25.858530 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.856293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret podName:3d6a461d-3472-4285-b5a9-1a11367f6497 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:29.856282452 +0000 UTC m=+10.149152179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret") pod "global-pull-secret-syncer-cjhm7" (UID: "3d6a461d-3472-4285-b5a9-1a11367f6497") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:25.956502 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:25.956366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:25.956728 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.956540 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:25.956728 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.956564 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:25.956728 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.956579 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vl7nn for pod openshift-network-diagnostics/network-check-target-gxvcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:25.956728 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:25.956647 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn podName:969e817f-6f19-4c15-835c-abcb599d5554 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:29.956628075 +0000 UTC m=+10.249497801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl7nn" (UniqueName: "kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn") pod "network-check-target-gxvcp" (UID: "969e817f-6f19-4c15-835c-abcb599d5554") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:26.186954 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:26.186925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:26.187105 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:26.187081 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:26.187569 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:26.187549 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:26.187693 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:26.187646 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:26.187837 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:26.187822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:26.187938 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:26.187905 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:26.244224 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:26.242385 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"7cf4b9ea98890a60ae37fd39b8c09d27","Type":"ContainerStarted","Data":"e523ccb9e140e301858aa3f2a634f3c1c1d52945b6992144640b68aac002e092"} Nov 04 03:16:26.250536 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:26.250032 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerStarted","Data":"0788dfffd7054f87d8f4dc24909e0cdb0f78f44c139acf4894fc6ee104e13fde"} Nov 04 03:16:26.362900 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:26.362344 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" podStartSLOduration=5.362327026 podStartE2EDuration="5.362327026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:26.361567502 +0000 UTC m=+6.654437237" watchObservedRunningTime="2026-11-04 03:16:26.362327026 +0000 UTC m=+6.655196762" Nov 04 03:16:27.265512 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:27.265314 2579 generic.go:356] "Generic (PLEG): container finished" podID="ee48ef9881478a9b0ef921b39f5cf13c" containerID="e31c21a72b201f4a52d25b3826381397a898d490ca70dde55531f58eaae4f390" exitCode=0 Nov 04 03:16:27.265512 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:27.265419 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"ee48ef9881478a9b0ef921b39f5cf13c","Type":"ContainerDied","Data":"e31c21a72b201f4a52d25b3826381397a898d490ca70dde55531f58eaae4f390"} Nov 04 03:16:27.271288 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:27.269933 2579 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="0788dfffd7054f87d8f4dc24909e0cdb0f78f44c139acf4894fc6ee104e13fde" exitCode=0 Nov 04 03:16:27.271288 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:27.270835 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerDied","Data":"0788dfffd7054f87d8f4dc24909e0cdb0f78f44c139acf4894fc6ee104e13fde"} Nov 04 03:16:28.187815 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.187156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:28.187815 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:28.187281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:28.187815 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.187354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:28.187815 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:28.187434 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:28.187815 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.187499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:28.187815 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:28.187567 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:28.274095 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.274020 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"ee48ef9881478a9b0ef921b39f5cf13c","Type":"ContainerStarted","Data":"8f7b2106a52f2c4b32d6e9939a3b841e5739682c6a0623c5e1f903c28e5aae94"} Nov 04 03:16:28.277940 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.277512 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5xngn" event={"ID":"dab481c8-cff5-4c29-aa52-552434b043bf","Type":"ContainerStarted","Data":"32a0cf1e4a23658999aeca0e1b902e0669402c1c305e97ff680bfe842da170c0"} Nov 04 03:16:28.280014 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.279626 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9jbx" event={"ID":"e4b5da75-9530-4e0b-849d-fa50fdec3241","Type":"ContainerStarted","Data":"fbfb96bd75cab60f33ce51680521ab3a16ca8f74374b29ab1086b030dad707c8"} Nov 04 03:16:28.506046 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.505567 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" podStartSLOduration=7.505552943 podStartE2EDuration="7.505552943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:28.390580939 +0000 UTC m=+8.683450675" watchObservedRunningTime="2026-11-04 03:16:28.505552943 +0000 UTC m=+8.798422679" Nov 04 03:16:28.633945 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:28.633116 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9jbx" podStartSLOduration=3.170851158 podStartE2EDuration="8.633093621s" totalImagesPullingTime="5.462242463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:28.505852436 +0000 UTC m=+8.798722173" watchObservedRunningTime="2026-11-04 03:16:28.633093621 +0000 UTC m=+8.925963356" Nov 04 03:16:29.895146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:29.894819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:29.895146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:29.894898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:29.895146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.895037 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:29.895146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.895093 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret podName:3d6a461d-3472-4285-b5a9-1a11367f6497 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:37.895080237 +0000 UTC m=+18.187949963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret") pod "global-pull-secret-syncer-cjhm7" (UID: "3d6a461d-3472-4285-b5a9-1a11367f6497") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:29.895809 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.895398 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:29.895809 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.895429 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:16:37.895420032 +0000 UTC m=+18.188289758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:29.995429 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:29.995388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:29.995626 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.995616 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:29.995701 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.995634 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:29.995701 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.995646 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vl7nn for pod openshift-network-diagnostics/network-check-target-gxvcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:29.995701 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:29.995698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn podName:969e817f-6f19-4c15-835c-abcb599d5554 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:37.995678664 +0000 UTC m=+18.288548390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl7nn" (UniqueName: "kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn") pod "network-check-target-gxvcp" (UID: "969e817f-6f19-4c15-835c-abcb599d5554") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:30.186154 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:30.186080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:30.186315 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:30.186226 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:30.186954 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:30.186692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:30.186954 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:30.186767 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:30.186954 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:30.186864 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:30.186954 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:30.186912 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:30.287969 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:30.287924 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" event={"ID":"3e9891bc-c81c-4ce5-afc9-75c07b82176a","Type":"ContainerStarted","Data":"535f37be122f6ad2074d24e043213a85e73412c133fa825c9008b15108c4e864"} Nov 04 03:16:31.292252 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.292223 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7ztjl" event={"ID":"d4132467-8852-45ae-8c94-bf0d78847bc6","Type":"ContainerStarted","Data":"699e1372fd2701b06d6e44300812d97a7a17cf1eb2886229004080d63f8a1241"} Nov 04 03:16:31.295392 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.295369 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-svnhr" event={"ID":"9380c359-acb2-43f6-af23-89baa5bee3b4","Type":"ContainerStarted","Data":"d1779f0fe92e3b6a51a754185668cdf425b567a0e285b1d7c17309e0af4102f1"} Nov 04 03:16:31.332579 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.332425 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 03:16:31.407534 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.407458 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5xngn" podStartSLOduration=6.728443819 podStartE2EDuration="11.407442644s" totalImagesPullingTime="4.678998825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:28.637627629 +0000 UTC m=+8.930497367" watchObservedRunningTime="2026-11-04 03:16:31.407442644 +0000 UTC m=+11.700312376" Nov 04 03:16:31.528041 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.527730 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-7ztjl" podStartSLOduration=3.190289747 podStartE2EDuration="11.527700887s" totalImagesPullingTime="8.33741114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:31.408258303 +0000 UTC m=+11.701128038" watchObservedRunningTime="2026-11-04 03:16:31.527700887 +0000 UTC m=+11.820570627" Nov 04 03:16:31.800346 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.800104 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:31.801057 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.801036 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:31.921524 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:31.921443 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-svnhr" podStartSLOduration=3.704713647 podStartE2EDuration="11.921422612s" totalImagesPullingTime="8.216708965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:31.528025954 +0000 UTC m=+11.820895690" watchObservedRunningTime="2026-11-04 03:16:31.921422612 +0000 UTC m=+12.214292348" Nov 04 03:16:32.185462 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.185393 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:32.185601 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:32.185520 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:32.185880 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.185859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:32.185993 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:32.185965 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:32.186091 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.186025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:32.186091 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:32.186076 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:32.195972 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.195879 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T03:16:31.332442624Z","UUID":"74f29c91-1c8b-412c-bae7-68fa9a481d6f","Handler":null,"Name":"","Endpoint":""} Nov 04 03:16:32.197605 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.197587 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 03:16:32.197687 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.197617 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 03:16:32.299451 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.299407 2579 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="31683b328fc0139a324549e47b7db8e387b0438f9341f918d6416e71896a8bdd" exitCode=0 Nov 04 03:16:32.299957 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.299518 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerDied","Data":"31683b328fc0139a324549e47b7db8e387b0438f9341f918d6416e71896a8bdd"} Nov 04 03:16:32.302658 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:32.302572 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" event={"ID":"3e9891bc-c81c-4ce5-afc9-75c07b82176a","Type":"ContainerStarted","Data":"e1486b5c3701e53351d853ea6a58cb9d0f0777ae446cd1e2c7dd2419735918d6"} Nov 04 03:16:33.306690 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:33.306617 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" event={"ID":"3e9891bc-c81c-4ce5-afc9-75c07b82176a","Type":"ContainerStarted","Data":"867d9b20ffb0f739f184f8be7cad003d90f2ecb8b6d9497ef0e2c27138bcc880"} Nov 04 03:16:33.308387 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:33.308330 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" event={"ID":"b24d1578-2e1c-4602-a90e-1d86e1b10645","Type":"ContainerStarted","Data":"f5b0f19dedc2d1a84a6b424312e167c5afc8ff806cb09ce84df2bc441ca03cae"} Nov 04 03:16:33.310087 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:33.310066 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khw7x" event={"ID":"e04fdd96-e8ac-4338-933c-1b9dadb56b89","Type":"ContainerStarted","Data":"eed6531cc61175998877c9ce32734405f16cdee0eb0827bedceb0e4e12064c84"} Nov 04 03:16:33.425029 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:33.424979 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" podStartSLOduration=3.459338769 podStartE2EDuration="13.424965972s" totalImagesPullingTime="9.965627203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:33.424756991 +0000 UTC m=+13.717626732" watchObservedRunningTime="2026-11-04 03:16:33.424965972 +0000 UTC m=+13.717835707" Nov 04 03:16:33.543391 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:33.543338 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" podStartSLOduration=3.031270949 podStartE2EDuration="13.543320101s" totalImagesPullingTime="10.512049152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:33.543058746 +0000 UTC m=+13.835928484" watchObservedRunningTime="2026-11-04 03:16:33.543320101 +0000 UTC m=+13.836189838" Nov 04 03:16:33.658005 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:33.657958 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-khw7x" podStartSLOduration=4.976986668 podStartE2EDuration="13.657944089s" totalImagesPullingTime="8.680957421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:33.65716178 +0000 UTC m=+13.950031517" watchObservedRunningTime="2026-11-04 03:16:33.657944089 +0000 UTC m=+13.950813824" Nov 04 03:16:34.185390 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:34.185351 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:34.185568 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:34.185353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:34.185568 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:34.185492 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:34.185568 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:34.185502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:34.185716 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:34.185590 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:34.185716 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:34.185670 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:34.315206 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:34.315175 2579 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="0a5c4e8480af28ac9dbaa270d660c8465f9dc8f630da117792a4f8d19bf4277e" exitCode=0 Nov 04 03:16:34.315907 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:34.315297 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerDied","Data":"0a5c4e8480af28ac9dbaa270d660c8465f9dc8f630da117792a4f8d19bf4277e"} Nov 04 03:16:36.185589 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:36.185393 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:36.186016 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:36.185400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:36.186016 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:36.185694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:36.186016 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:36.185462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:36.186016 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:36.185776 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:36.186016 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:36.185866 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:36.322377 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:36.322289 2579 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="74494edcc789da987900f79efa69514134fdf57fcde75fc493d1290e951c83fa" exitCode=0 Nov 04 03:16:36.322377 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:36.322350 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerDied","Data":"74494edcc789da987900f79efa69514134fdf57fcde75fc493d1290e951c83fa"} Nov 04 03:16:37.965035 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:37.964851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:37.965355 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:37.965081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:37.965355 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:37.965004 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:37.965355 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:37.965153 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret podName:3d6a461d-3472-4285-b5a9-1a11367f6497 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:53.965138823 +0000 UTC m=+34.258008550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret") pod "global-pull-secret-syncer-cjhm7" (UID: "3d6a461d-3472-4285-b5a9-1a11367f6497") : object "kube-system"/"original-pull-secret" not registered Nov 04 03:16:37.965355 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:37.965167 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:37.965355 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:37.965205 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:16:53.965194444 +0000 UTC m=+34.258064157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 04 03:16:38.065742 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.065502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:38.065742 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.065635 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 04 03:16:38.065742 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.065650 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 04 03:16:38.065742 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.065661 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vl7nn for pod openshift-network-diagnostics/network-check-target-gxvcp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:38.065742 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.065712 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn podName:969e817f-6f19-4c15-835c-abcb599d5554 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:54.065696554 +0000 UTC m=+34.358566271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vl7nn" (UniqueName: "kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn") pod "network-check-target-gxvcp" (UID: "969e817f-6f19-4c15-835c-abcb599d5554") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 04 03:16:38.185441 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.185347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:38.185640 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.185347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:38.185640 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.185494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:38.185640 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.185493 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:38.185640 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.185589 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:38.185855 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:38.185661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:38.329801 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.329770 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:16:38.330194 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330158 2579 generic.go:356] "Generic (PLEG): container finished" podID="c66837ee-2658-4ef6-bee5-d15bbe58ebde" containerID="89dade8b81d5c89823e9db759af485d2576466aa708bd258528af613b0caaab6" exitCode=1 Nov 04 03:16:38.330299 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330215 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"a5d47865bf4cad1be697a6f85f1d4c0ee84bc31bd6544230e6a0787707e81040"} Nov 04 03:16:38.330299 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330289 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"df7513fd51ad4118a39b81ea45594e55cab2d9246efc3205e856136f60ca2849"} Nov 04 03:16:38.330415 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330338 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"8a6f1306081fef72fbd36720b7bf9ec0cda0c2150c4d02514c41f6121da3e9ba"} Nov 04 03:16:38.330415 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330384 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"885f5e41303f8637c4656eb2daa447f668fe5952a437c3b45d2f1df7b4fd51ff"} Nov 04 03:16:38.330537 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330431 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerDied","Data":"89dade8b81d5c89823e9db759af485d2576466aa708bd258528af613b0caaab6"} Nov 04 03:16:38.330537 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:38.330505 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"dd07cccc91b65d6040456c1e2a552eac6cd16e4cb5c5390b0e0406cac9aecb7b"} Nov 04 03:16:39.334785 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:39.334750 2579 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="60d179391a7df18d1cbcb299403477105cb4d62174815ce873f86ae0c326e2a7" exitCode=0 Nov 04 03:16:39.335323 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:39.334811 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerDied","Data":"60d179391a7df18d1cbcb299403477105cb4d62174815ce873f86ae0c326e2a7"} Nov 04 03:16:40.186292 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.186258 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:40.186429 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.186300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:40.186429 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.186392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:40.186429 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:40.186394 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:40.186566 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:40.186486 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:40.186566 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:40.186534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:40.340371 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.340347 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:16:40.341000 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.340771 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"23c9608a75a1997583275254248ca90a0d09c643ff57cc9e066d8c71d1d5e4d5"} Nov 04 03:16:40.343688 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.343668 2579 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="2d307c561a738723d6d4eb2201b580ee003e6c277f75c91fc299a6fb8b256e87" exitCode=0 Nov 04 03:16:40.343793 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:40.343703 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerDied","Data":"2d307c561a738723d6d4eb2201b580ee003e6c277f75c91fc299a6fb8b256e87"} Nov 04 03:16:41.349040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:41.349000 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" event={"ID":"284dd9ee-7ea1-4c8d-9674-57c15be65836","Type":"ContainerStarted","Data":"33b734d899a5dddfb4266071d555c9b645eeed8427c3d9f8d278f56c9dea60a7"} Nov 04 03:16:41.470938 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:41.470889 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j2p7f" podStartSLOduration=8.886827777 podStartE2EDuration="21.470875242s" totalImagesPullingTime="12.584047465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:41.470844243 +0000 UTC m=+21.763713977" watchObservedRunningTime="2026-11-04 03:16:41.470875242 +0000 UTC m=+21.763744977" Nov 04 03:16:42.185917 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:42.185888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:42.186049 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:42.185945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:42.186049 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:42.186025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:42.186172 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:42.186149 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:42.186264 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:42.186247 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:42.186349 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:42.186328 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:42.354990 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:42.354816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:16:42.355492 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:42.355373 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"bcd6e0d7c92810326f88e8c3def79db882247426548a483c6fb3a718baa8f8f4"} Nov 04 03:16:42.356107 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:42.356088 2579 scope.go:122] "RemoveContainer" containerID="89dade8b81d5c89823e9db759af485d2576466aa708bd258528af613b0caaab6" Nov 04 03:16:43.360488 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.360449 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:16:43.360949 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.360816 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" event={"ID":"c66837ee-2658-4ef6-bee5-d15bbe58ebde","Type":"ContainerStarted","Data":"20aaadb9c967541b3df294b681764b4edf0238bb1cb9ce2eb4be0399b698abff"} Nov 04 03:16:43.361071 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.361005 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:43.361126 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.361078 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:43.361168 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.361128 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:43.377448 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.377422 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:43.377825 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.377808 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:16:43.586943 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.586906 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:43.587122 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.587043 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:16:43.587485 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.587441 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:16:43.616449 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:43.616351 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" podStartSLOduration=8.560348484 podStartE2EDuration="23.616337918s" totalImagesPullingTime="15.055989434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:43.589927484 +0000 UTC m=+23.882797229" watchObservedRunningTime="2026-11-04 03:16:43.616337918 +0000 UTC m=+23.909207660" Nov 04 03:16:44.185917 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.185879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:44.185917 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.185913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:44.186104 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.185991 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:44.186166 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:44.186107 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:44.186226 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:44.186204 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:44.186312 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:44.186296 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:44.898015 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.897980 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cjhm7"] Nov 04 03:16:44.898384 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.898094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:44.898384 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:44.898181 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:44.900581 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.900551 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gxvcp"] Nov 04 03:16:44.900698 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.900637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:44.900741 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:44.900714 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:44.903969 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.903948 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m9tdv"] Nov 04 03:16:44.904162 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:44.904146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:44.904274 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:44.904255 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:46.184899 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:46.184866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:46.185356 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:46.184866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:46.185356 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:46.184974 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:46.185356 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:46.185090 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:46.185356 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:46.185142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:46.185356 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:46.185227 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:48.185774 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:48.185734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:48.185774 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:48.185752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:48.186225 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:48.185854 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:48.186225 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:48.185867 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:48.186225 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:48.185939 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:48.186225 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:48.185987 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:50.186659 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:50.186450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:50.187101 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:50.186525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:50.187101 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:50.186723 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gxvcp" podUID="969e817f-6f19-4c15-835c-abcb599d5554" Nov 04 03:16:50.187101 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:50.186544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:50.187101 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:50.186816 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tdv" podUID="074a2389-db4e-4779-979e-0b9ba07a65bf" Nov 04 03:16:50.187101 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:50.186926 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cjhm7" podUID="3d6a461d-3472-4285-b5a9-1a11367f6497" Nov 04 03:16:51.060349 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.060326 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeReady" Nov 04 03:16:51.060568 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.060442 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 04 03:16:51.176794 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.176758 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g5nm2"] Nov 04 03:16:51.181428 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.181402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:16:51.225461 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.225429 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7n6xm"] Nov 04 03:16:51.228382 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.228355 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk"] Nov 04 03:16:51.228544 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.228521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:16:51.231346 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.231329 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n2zsd"] Nov 04 03:16:51.231464 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.231446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.234234 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.234212 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9sfxr"] Nov 04 03:16:51.234348 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.234332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:16:51.237040 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.237025 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f89dfb67b-5spht"] Nov 04 03:16:51.237150 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.237137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.240642 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.240624 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp"] Nov 04 03:16:51.240744 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.240725 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.243332 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.243313 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz"] Nov 04 03:16:51.243433 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.243419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:16:51.246189 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.246171 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5cxt"] Nov 04 03:16:51.246291 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.246275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.248994 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.248976 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7685ff7ccc-rbhsz"] Nov 04 03:16:51.249101 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.249087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.252113 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.252094 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.280123 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.275939 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.280123 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.276505 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66bfccb89c-84k2x"] Nov 04 03:16:51.280406 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.280379 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.282172 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.281223 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kx5dm"] Nov 04 03:16:51.282172 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.281546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.284199 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284181 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg"] Nov 04 03:16:51.284331 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284310 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:51.284394 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284343 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.284453 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284396 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 03:16:51.284547 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284494 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.284609 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284560 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 03:16:51.284609 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.284601 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xgt9d\"" Nov 04 03:16:51.287334 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.287316 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg"] Nov 04 03:16:51.287437 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.287425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.295768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.295740 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj"] Nov 04 03:16:51.296176 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.295912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.301657 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.299812 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zh4vg"] Nov 04 03:16:51.301657 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.299934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:16:51.302750 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.302725 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8cvwl"] Nov 04 03:16:51.302890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.302874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.306229 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306205 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g5nm2"] Nov 04 03:16:51.306326 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306244 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7n6xm"] Nov 04 03:16:51.306326 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306265 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk"] Nov 04 03:16:51.306326 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306293 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9sfxr"] Nov 04 03:16:51.306326 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.306495 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306312 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz"] Nov 04 03:16:51.306495 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306460 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5cxt"] Nov 04 03:16:51.306583 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.306503 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f89dfb67b-5spht"] Nov 04 03:16:51.326514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.326494 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7685ff7ccc-rbhsz"] Nov 04 03:16:51.340940 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.340913 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kx5dm"] Nov 04 03:16:51.343788 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.343759 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n2zsd"] Nov 04 03:16:51.343900 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.343822 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg"] Nov 04 03:16:51.344764 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.344742 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8cvwl"] Nov 04 03:16:51.345185 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345166 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.345562 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345543 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.345783 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345763 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.345865 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345817 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.345865 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345828 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.345865 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345851 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.346027 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345886 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 03:16:51.346027 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345979 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.346027 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346010 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.346148 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346063 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.346148 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346080 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.346148 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346117 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 03:16:51.346148 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346126 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 03:16:51.346327 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346181 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 03:16:51.346327 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346222 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.346327 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.345980 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.346327 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346315 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 03:16:51.346434 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.346352 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 03:16:51.353836 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.353814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed988d6-f52e-4030-8805-1c214876ee88-serving-cert\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.353928 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.353851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.353928 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.353900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.353994 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.353959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.353994 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.353987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.354059 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.354090 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.354090 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.354147 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-config\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.354218 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.354218 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.354218 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.354302 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-snapshots\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.354302 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.354302 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-stats-auth\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.354404 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkz8\" (UniqueName: \"kubernetes.io/projected/f762673f-00d6-4182-b7e3-bd16847fd218-kube-api-access-wgkz8\") pod \"kas-connection-checker-685895c7c9-7n6xm\" (UID: \"f762673f-00d6-4182-b7e3-bd16847fd218\") " pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:16:51.354404 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-tmp\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.354404 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.354523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm9p\" (UniqueName: \"kubernetes.io/projected/6802ea74-e881-496d-b63b-601d10ae7d02-kube-api-access-srm9p\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.354523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0cabe74-608b-461e-a3aa-fa049a28de5b-serving-cert\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.354523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-default-certificate\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.354603 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.354603 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.354662 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74fmz\" (UniqueName: \"kubernetes.io/projected/a699a906-fae8-47d9-92ba-8eefce6cf724-kube-api-access-74fmz\") pod \"kas-connection-checker-685895c7c9-n2zsd\" (UID: \"a699a906-fae8-47d9-92ba-8eefce6cf724\") " pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:16:51.354662 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86941193-b41d-47a0-b4cf-ef4221272e45-config\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.354662 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7dg\" (UniqueName: \"kubernetes.io/projected/0ed988d6-f52e-4030-8805-1c214876ee88-kube-api-access-pf7dg\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.354746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86941193-b41d-47a0-b4cf-ef4221272e45-serving-cert\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.354746 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnts4\" (UniqueName: \"kubernetes.io/projected/d0cabe74-608b-461e-a3aa-fa049a28de5b-kube-api-access-vnts4\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.354803 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.354803 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.354803 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.354886 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.354886 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l5sg\" (UniqueName: \"kubernetes.io/projected/781b84f6-581f-4357-b447-8d4f92aece81-kube-api-access-4l5sg\") pod \"kas-connection-checker-685895c7c9-g5nm2\" (UID: \"781b84f6-581f-4357-b447-8d4f92aece81\") " pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:16:51.354886 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.354970 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.355000 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2hs\" (UniqueName: \"kubernetes.io/projected/86941193-b41d-47a0-b4cf-ef4221272e45-kube-api-access-rc2hs\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.355031 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.354999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.355079 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.355027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2979t\" (UniqueName: \"kubernetes.io/projected/3194fa09-26b5-46ff-bada-c1077f024432-kube-api-access-2979t\") pod \"volume-data-source-validator-58c9f5f46d-pnvkp\" (UID: \"3194fa09-26b5-46ff-bada-c1077f024432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:16:51.355079 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.355051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.355179 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.355083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.355179 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.355132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-487vl\" (UniqueName: \"kubernetes.io/projected/ad24bd15-2732-4e05-9494-ca0691fd9e86-kube-api-access-487vl\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.398150 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.398117 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 03:16:51.398541 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.398516 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 03:16:51.398738 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.398713 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 03:16:51.398809 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.398753 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 03:16:51.426386 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.426350 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 03:16:51.426555 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.426386 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 03:16:51.426555 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.426428 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 03:16:51.426555 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.426389 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 03:16:51.427239 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.427222 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66bfccb89c-84k2x"] Nov 04 03:16:51.438276 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438251 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 03:16:51.438369 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438264 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 03:16:51.438369 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438274 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 03:16:51.438369 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438342 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 03:16:51.438505 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438394 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n89nd\"" Nov 04 03:16:51.438505 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438460 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-57t69\"" Nov 04 03:16:51.438721 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438706 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 03:16:51.438778 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438753 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 03:16:51.438827 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438796 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 03:16:51.438827 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.438802 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 03:16:51.439296 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439275 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6wxff\"" Nov 04 03:16:51.439296 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439271 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4sx77\"" Nov 04 03:16:51.439459 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439315 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 03:16:51.439459 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439354 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ccs79\"" Nov 04 03:16:51.439579 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439538 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 03:16:51.440282 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439733 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2qrjk\"" Nov 04 03:16:51.440282 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439737 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2c56k\"" Nov 04 03:16:51.440282 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439823 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r8flv\"" Nov 04 03:16:51.440282 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439861 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x7rx6\"" Nov 04 03:16:51.440282 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.439929 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mprmh\"" Nov 04 03:16:51.443820 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.443802 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zh4vg"] Nov 04 03:16:51.444440 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.444422 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp"] Nov 04 03:16:51.445051 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445036 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg"] Nov 04 03:16:51.445125 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445110 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.445176 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445137 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.445176 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445148 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:16:51.445176 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445151 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.445176 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445173 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 03:16:51.445316 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445189 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.445316 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445220 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 03:16:51.445686 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.445672 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj"] Nov 04 03:16:51.456309 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.456354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.456354 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.456439 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/986942d0-73ec-4567-9026-48482bd186ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.456439 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056b218a-8def-4ac5-8a9b-78b69b08943d-cert\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.456439 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/63620800-28b7-4d31-88d8-b75f34a5dda1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.456439 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-snapshots\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.456595 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.456448 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:16:51.456595 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.456595 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.456545 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.956525755 +0000 UTC m=+32.249395468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:16:51.456595 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-stats-auth\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.456748 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.456611 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:51.456748 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkz8\" (UniqueName: \"kubernetes.io/projected/f762673f-00d6-4182-b7e3-bd16847fd218-kube-api-access-wgkz8\") pod \"kas-connection-checker-685895c7c9-7n6xm\" (UID: \"f762673f-00d6-4182-b7e3-bd16847fd218\") " pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:16:51.456748 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.456626 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:16:51.456748 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.456674 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.956658103 +0000 UTC m=+32.249527816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:16:51.456897 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnz4\" (UniqueName: \"kubernetes.io/projected/63620800-28b7-4d31-88d8-b75f34a5dda1-kube-api-access-htnz4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.456897 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-tmp\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.456987 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.456987 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm9p\" (UniqueName: \"kubernetes.io/projected/6802ea74-e881-496d-b63b-601d10ae7d02-kube-api-access-srm9p\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.456987 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.456954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.457132 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0cabe74-608b-461e-a3aa-fa049a28de5b-serving-cert\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.457132 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-snapshots\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.457231 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-default-certificate\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.457231 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-tmp\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.457231 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.457363 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.457363 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.457363 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:51.457363 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fmz\" (UniqueName: \"kubernetes.io/projected/a699a906-fae8-47d9-92ba-8eefce6cf724-kube-api-access-74fmz\") pod \"kas-connection-checker-685895c7c9-n2zsd\" (UID: \"a699a906-fae8-47d9-92ba-8eefce6cf724\") " pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:16:51.457363 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86941193-b41d-47a0-b4cf-ef4221272e45-config\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.457592 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7dg\" (UniqueName: \"kubernetes.io/projected/0ed988d6-f52e-4030-8805-1c214876ee88-kube-api-access-pf7dg\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.457592 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457401 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.457592 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86941193-b41d-47a0-b4cf-ef4221272e45-serving-cert\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.457592 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnts4\" (UniqueName: \"kubernetes.io/projected/d0cabe74-608b-461e-a3aa-fa049a28de5b-kube-api-access-vnts4\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.457831 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.457877 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:51.457919 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.457962 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.457962 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.458050 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.457983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5z2j\" (UniqueName: \"kubernetes.io/projected/939eba33-05f1-4a74-97e4-3c41074ca9ac-kube-api-access-t5z2j\") pod \"network-check-source-6f78d59f99-27kkj\" (UID: \"939eba33-05f1-4a74-97e4-3c41074ca9ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:16:51.458050 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-tmp-dir\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.458050 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.458178 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63620800-28b7-4d31-88d8-b75f34a5dda1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.458178 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5sg\" (UniqueName: \"kubernetes.io/projected/781b84f6-581f-4357-b447-8d4f92aece81-kube-api-access-4l5sg\") pod \"kas-connection-checker-685895c7c9-g5nm2\" (UID: \"781b84f6-581f-4357-b447-8d4f92aece81\") " pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:16:51.458178 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458130 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.458178 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86941193-b41d-47a0-b4cf-ef4221272e45-config\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.458178 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.458409 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2hs\" (UniqueName: \"kubernetes.io/projected/86941193-b41d-47a0-b4cf-ef4221272e45-kube-api-access-rc2hs\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.458409 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.458409 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2979t\" (UniqueName: \"kubernetes.io/projected/3194fa09-26b5-46ff-bada-c1077f024432-kube-api-access-2979t\") pod \"volume-data-source-validator-58c9f5f46d-pnvkp\" (UID: \"3194fa09-26b5-46ff-bada-c1077f024432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:16:51.458588 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.458588 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.458687 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.458612 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.958595123 +0000 UTC m=+32.251464855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:16:51.458687 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487vl\" (UniqueName: \"kubernetes.io/projected/ad24bd15-2732-4e05-9494-ca0691fd9e86-kube-api-access-487vl\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.458687 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4clk\" (UniqueName: \"kubernetes.io/projected/056b218a-8def-4ac5-8a9b-78b69b08943d-kube-api-access-m4clk\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.458837 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed988d6-f52e-4030-8805-1c214876ee88-serving-cert\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.458837 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-config-volume\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.458988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.459113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.459579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.459818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.459824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.460246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.460559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.460676 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.460743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.460779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.460845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2z99\" (UniqueName: \"kubernetes.io/projected/986942d0-73ec-4567-9026-48482bd186ae-kube-api-access-j2z99\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.460929 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.460972 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.960958743 +0000 UTC m=+32.253828457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.461024 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.461034 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:16:51.461146 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.461066 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:51.961056499 +0000 UTC m=+32.253926212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhwc\" (UniqueName: \"kubernetes.io/projected/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-kube-api-access-tkhwc\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-config\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461682 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461901 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86941193-b41d-47a0-b4cf-ef4221272e45-serving-cert\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.461934 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461913 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-config\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.462348 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.461998 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0cabe74-608b-461e-a3aa-fa049a28de5b-serving-cert\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.462348 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.462277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.462462 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.462406 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.462546 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.462514 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-default-certificate\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.463670 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.463649 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed988d6-f52e-4030-8805-1c214876ee88-serving-cert\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.463763 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.463747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-stats-auth\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.464158 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.464138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.518231 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.518195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkz8\" (UniqueName: \"kubernetes.io/projected/f762673f-00d6-4182-b7e3-bd16847fd218-kube-api-access-wgkz8\") pod \"kas-connection-checker-685895c7c9-7n6xm\" (UID: \"f762673f-00d6-4182-b7e3-bd16847fd218\") " pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:16:51.518352 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.518249 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.518413 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.518393 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm9p\" (UniqueName: \"kubernetes.io/projected/6802ea74-e881-496d-b63b-601d10ae7d02-kube-api-access-srm9p\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.525075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.525043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5sg\" (UniqueName: \"kubernetes.io/projected/781b84f6-581f-4357-b447-8d4f92aece81-kube-api-access-4l5sg\") pod \"kas-connection-checker-685895c7c9-g5nm2\" (UID: \"781b84f6-581f-4357-b447-8d4f92aece81\") " pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:16:51.525437 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.525410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnts4\" (UniqueName: \"kubernetes.io/projected/d0cabe74-608b-461e-a3aa-fa049a28de5b-kube-api-access-vnts4\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.525631 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.525605 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fmz\" (UniqueName: \"kubernetes.io/projected/a699a906-fae8-47d9-92ba-8eefce6cf724-kube-api-access-74fmz\") pod \"kas-connection-checker-685895c7c9-n2zsd\" (UID: \"a699a906-fae8-47d9-92ba-8eefce6cf724\") " pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:16:51.526002 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.525976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487vl\" (UniqueName: \"kubernetes.io/projected/ad24bd15-2732-4e05-9494-ca0691fd9e86-kube-api-access-487vl\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.527146 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.527109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2hs\" (UniqueName: \"kubernetes.io/projected/86941193-b41d-47a0-b4cf-ef4221272e45-kube-api-access-rc2hs\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.527244 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.527186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.527293 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.527267 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7dg\" (UniqueName: \"kubernetes.io/projected/0ed988d6-f52e-4030-8805-1c214876ee88-kube-api-access-pf7dg\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.527621 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.527599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2979t\" (UniqueName: \"kubernetes.io/projected/3194fa09-26b5-46ff-bada-c1077f024432-kube-api-access-2979t\") pod \"volume-data-source-validator-58c9f5f46d-pnvkp\" (UID: \"3194fa09-26b5-46ff-bada-c1077f024432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:16:51.527881 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.527864 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.527938 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.527922 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.540891 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.540868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:16:51.552170 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.552152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:16:51.557919 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.557901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:16:51.562054 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:51.562112 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5z2j\" (UniqueName: \"kubernetes.io/projected/939eba33-05f1-4a74-97e4-3c41074ca9ac-kube-api-access-t5z2j\") pod \"network-check-source-6f78d59f99-27kkj\" (UID: \"939eba33-05f1-4a74-97e4-3c41074ca9ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:16:51.562151 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-tmp-dir\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.562186 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63620800-28b7-4d31-88d8-b75f34a5dda1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.562230 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4clk\" (UniqueName: \"kubernetes.io/projected/056b218a-8def-4ac5-8a9b-78b69b08943d-kube-api-access-m4clk\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.562384 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-config-volume\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.562483 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z99\" (UniqueName: \"kubernetes.io/projected/986942d0-73ec-4567-9026-48482bd186ae-kube-api-access-j2z99\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.562552 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-tmp-dir\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.562613 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhwc\" (UniqueName: \"kubernetes.io/projected/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-kube-api-access-tkhwc\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.562677 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/986942d0-73ec-4567-9026-48482bd186ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.562677 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056b218a-8def-4ac5-8a9b-78b69b08943d-cert\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.562677 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/63620800-28b7-4d31-88d8-b75f34a5dda1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.562841 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnz4\" (UniqueName: \"kubernetes.io/projected/63620800-28b7-4d31-88d8-b75f34a5dda1-kube-api-access-htnz4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.562841 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.562841 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.562841 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:51.563035 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.562935 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:16:51.563035 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.562934 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-config-volume\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.563035 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.562992 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.062974849 +0000 UTC m=+32.355844577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:16:51.563199 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.563059 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:51.563199 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.563058 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:16:51.563199 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.563096 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.063084173 +0000 UTC m=+32.355953902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:51.563199 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.563125 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.063106656 +0000 UTC m=+32.355976370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:16:51.563667 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.563644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:51.563745 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.563673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/63620800-28b7-4d31-88d8-b75f34a5dda1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.563745 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.563689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/986942d0-73ec-4567-9026-48482bd186ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.565348 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.565320 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63620800-28b7-4d31-88d8-b75f34a5dda1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.565811 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.565789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056b218a-8def-4ac5-8a9b-78b69b08943d-cert\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.569907 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.569879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:16:51.574849 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.574826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:16:51.584147 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.584119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:16:51.627869 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.627684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhwc\" (UniqueName: \"kubernetes.io/projected/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-kube-api-access-tkhwc\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:51.634770 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.634735 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnz4\" (UniqueName: \"kubernetes.io/projected/63620800-28b7-4d31-88d8-b75f34a5dda1-kube-api-access-htnz4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.636826 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.636796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z99\" (UniqueName: \"kubernetes.io/projected/986942d0-73ec-4567-9026-48482bd186ae-kube-api-access-j2z99\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:51.650253 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.650184 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4clk\" (UniqueName: \"kubernetes.io/projected/056b218a-8def-4ac5-8a9b-78b69b08943d-kube-api-access-m4clk\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.653203 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.653138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5z2j\" (UniqueName: \"kubernetes.io/projected/939eba33-05f1-4a74-97e4-3c41074ca9ac-kube-api-access-t5z2j\") pod \"network-check-source-6f78d59f99-27kkj\" (UID: \"939eba33-05f1-4a74-97e4-3c41074ca9ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:16:51.792719 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.792681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:16:51.917720 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.917688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:16:51.929759 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.929731 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:16:51.935779 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.935758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:16:51.942901 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.942866 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7n6xm"] Nov 04 03:16:51.946681 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.946626 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-9sfxr"] Nov 04 03:16:51.952768 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.951464 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-n2zsd"] Nov 04 03:16:51.967314 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.967279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.967415 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.967361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:51.967415 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.967389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:51.967520 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.967447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:51.967520 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:51.967509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:51.967662 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967647 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:51.967697 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967667 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:16:51.967743 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967730 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.967712921 +0000 UTC m=+33.260582636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:16:51.967816 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967802 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.967792022 +0000 UTC m=+33.260661749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:16:51.967878 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967867 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:51.967915 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967880 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:16:51.967915 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967908 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.967898897 +0000 UTC m=+33.260768628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:16:51.967979 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967967 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:16:51.968011 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.967996 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.967986826 +0000 UTC m=+33.260856554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:16:51.968062 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.968051 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:16:51.968096 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:51.968087 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:52.968076227 +0000 UTC m=+33.260945954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:16:52.013587 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.012929 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz"] Nov 04 03:16:52.013587 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.013003 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp"] Nov 04 03:16:52.022890 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.020759 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5cxt"] Nov 04 03:16:52.068272 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.068236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:52.068450 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.068293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:52.068450 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.068336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:52.068603 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.068458 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:16:52.068603 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.068546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:53.06852588 +0000 UTC m=+33.361395610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:16:52.068709 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.068615 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:16:52.068709 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.068646 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:53.068635381 +0000 UTC m=+33.361505111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:16:52.068709 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.068700 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:52.068860 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.068732 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:16:53.068721465 +0000 UTC m=+33.361591191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:52.175504 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.175419 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-g5nm2"] Nov 04 03:16:52.177496 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:52.177449 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781b84f6_581f_4357_b447_8d4f92aece81.slice/crio-e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c WatchSource:0}: Error finding container e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c: Status 404 returned error can't find the container with id e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c Nov 04 03:16:52.185014 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.184989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:52.185123 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.185013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:52.185123 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.185048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:52.238232 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.237856 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:16:52.238232 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.238039 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f24jt\"" Nov 04 03:16:52.238232 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.238100 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9qdq\"" Nov 04 03:16:52.238232 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.238156 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:16:52.238232 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.238208 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kfskp\"" Nov 04 03:16:52.308199 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.308169 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg"] Nov 04 03:16:52.310977 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:52.310947 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63620800_28b7_4d31_88d8_b75f34a5dda1.slice/crio-21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3 WatchSource:0}: Error finding container 21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3: Status 404 returned error can't find the container with id 21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3 Nov 04 03:16:52.320869 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.320849 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj"] Nov 04 03:16:52.323167 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:52.323146 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939eba33_05f1_4a74_97e4_3c41074ca9ac.slice/crio-b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d WatchSource:0}: Error finding container b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d: Status 404 returned error can't find the container with id b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d Nov 04 03:16:52.337682 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.337655 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zh4vg"] Nov 04 03:16:52.340333 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:52.340309 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056b218a_8def_4ac5_8a9b_78b69b08943d.slice/crio-ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c WatchSource:0}: Error finding container ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c: Status 404 returned error can't find the container with id ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c Nov 04 03:16:52.382514 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.382458 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zh4vg" event={"ID":"056b218a-8def-4ac5-8a9b-78b69b08943d","Type":"ContainerStarted","Data":"ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c"} Nov 04 03:16:52.383539 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.383509 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" event={"ID":"939eba33-05f1-4a74-97e4-3c41074ca9ac","Type":"ContainerStarted","Data":"b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d"} Nov 04 03:16:52.384613 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.384593 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" event={"ID":"63620800-28b7-4d31-88d8-b75f34a5dda1","Type":"ContainerStarted","Data":"21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3"} Nov 04 03:16:52.388521 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.388498 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" event={"ID":"a699a906-fae8-47d9-92ba-8eefce6cf724","Type":"ContainerStarted","Data":"a8bc40d7b58b524ffad195a83f64c34067570ccd16e81ed23f74e3b76a14fb60"} Nov 04 03:16:52.388602 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.388538 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" event={"ID":"a699a906-fae8-47d9-92ba-8eefce6cf724","Type":"ContainerStarted","Data":"c140d95316b4967461aa10a79a2e19041d9403334de8fc559103d94a68384fd5"} Nov 04 03:16:52.390115 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.390083 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" event={"ID":"781b84f6-581f-4357-b447-8d4f92aece81","Type":"ContainerStarted","Data":"15f511424d74daeb4d2f6268b549c8104f0edb3372d46675ca1fbebed566ac7f"} Nov 04 03:16:52.390198 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.390123 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" event={"ID":"781b84f6-581f-4357-b447-8d4f92aece81","Type":"ContainerStarted","Data":"e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c"} Nov 04 03:16:52.391242 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.391220 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" event={"ID":"0ed988d6-f52e-4030-8805-1c214876ee88","Type":"ContainerStarted","Data":"a0d2c34775ea931e4830a5f5b7667032aad241bd0cf5fa9ae068dd72b869874a"} Nov 04 03:16:52.392164 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.392146 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" event={"ID":"86941193-b41d-47a0-b4cf-ef4221272e45","Type":"ContainerStarted","Data":"d6f7f6313aa02a38a5ee7c17ebc47d1e5644e4e070e8b9b576e21574eca18c82"} Nov 04 03:16:52.393280 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.393262 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" event={"ID":"3194fa09-26b5-46ff-bada-c1077f024432","Type":"ContainerStarted","Data":"68892cba95d574b1321399d832b55986a6b2c7b1735a676da4ae55c4a5987ab9"} Nov 04 03:16:52.394282 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.394265 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" event={"ID":"d0cabe74-608b-461e-a3aa-fa049a28de5b","Type":"ContainerStarted","Data":"617a4896bb5a543bff4eb8a49ee5db024c7be3e8df5c290171e89a0d532e7f20"} Nov 04 03:16:52.395549 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.395532 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" event={"ID":"f762673f-00d6-4182-b7e3-bd16847fd218","Type":"ContainerStarted","Data":"65dcf6a16d2df45a15da3e8ec59f92ed538ef90bf552c1d367125845152dd72f"} Nov 04 03:16:52.395618 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.395561 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" event={"ID":"f762673f-00d6-4182-b7e3-bd16847fd218","Type":"ContainerStarted","Data":"94d498c93fdaba6a346e7e2eeda97dfc6acee5b5265f36f2dafa182a4d82db80"} Nov 04 03:16:52.975219 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.975190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:52.975409 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.975228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:52.975409 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975348 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:52.975409 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975367 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:16:52.975409 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.975377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:52.975409 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975412 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:54.975398252 +0000 UTC m=+35.268267992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.975431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975352 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975451 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975513 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:54.975506037 +0000 UTC m=+35.268375749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975527 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:54.975521541 +0000 UTC m=+35.268391254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975559 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975569 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975595 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:16:54.975583933 +0000 UTC m=+35.268453678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:16:52.975651 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:52.975610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:52.975940 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:52.975715 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:54.975688886 +0000 UTC m=+35.268558626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:16:53.076443 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.076406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:53.076626 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.076462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:53.076626 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.076518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:53.076626 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.076550 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:16:53.076626 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.076610 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:55.076593173 +0000 UTC m=+35.369462888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:16:53.076794 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.076637 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:53.076794 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.076677 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:16:53.076794 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.076704 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:16:55.076686577 +0000 UTC m=+35.369556303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:53.076794 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.076723 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:55.076713111 +0000 UTC m=+35.369582829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:16:53.484562 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.484501 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" podStartSLOduration=396.484460827 podStartE2EDuration="6m36.484460827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:53.364809978 +0000 UTC m=+33.657679713" watchObservedRunningTime="2026-11-04 03:16:53.484460827 +0000 UTC m=+33.777330563" Nov 04 03:16:53.601666 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.601601 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" podStartSLOduration=395.601581387 podStartE2EDuration="6m35.601581387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:53.600733338 +0000 UTC m=+33.893603073" watchObservedRunningTime="2026-11-04 03:16:53.601581387 +0000 UTC m=+33.894451124" Nov 04 03:16:53.602722 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.602677 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" podStartSLOduration=395.602662312 podStartE2EDuration="6m35.602662312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:53.484953915 +0000 UTC m=+33.777823667" watchObservedRunningTime="2026-11-04 03:16:53.602662312 +0000 UTC m=+33.895532051" Nov 04 03:16:53.934883 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.934818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:53.989571 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.989531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:53.990859 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.990812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:16:53.990984 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.990939 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:16:53.991031 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:53.991003 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:17:25.990984812 +0000 UTC m=+66.283854539 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : secret "metrics-daemon-secret" not found Nov 04 03:16:53.994042 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:53.994015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:54.015602 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.015569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:16:54.093921 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.092159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:54.100076 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.096414 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:54.218939 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.218759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:54.309730 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.309701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:16:54.411524 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.411495 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cjhm7"] Nov 04 03:16:54.520693 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.520662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:54.741777 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.741699 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gxvcp"] Nov 04 03:16:54.775223 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:54.775150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:54.807036 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:16:54.806897 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969e817f_6f19_4c15_835c_abcb599d5554.slice/crio-76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f WatchSource:0}: Error finding container 76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f: Status 404 returned error can't find the container with id 76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f Nov 04 03:16:55.003359 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.003283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:55.003359 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.003327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:55.003585 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.003375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:55.003585 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.003410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:55.003585 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.003561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:55.003762 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003728 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.003702352 +0000 UTC m=+39.296572086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:16:55.003827 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003816 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:16:55.003886 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003838 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:55.003886 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003850 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:16:55.003886 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003861 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.003847282 +0000 UTC m=+39.296716998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:16:55.003886 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003880 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.003870379 +0000 UTC m=+39.296740106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:16:55.004087 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003920 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:55.004087 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003931 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:16:55.004087 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.003961 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.003951086 +0000 UTC m=+39.296820802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:16:55.004087 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.004013 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:16:55.004087 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.004049 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.004039627 +0000 UTC m=+39.296909345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:16:55.034429 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.034312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.104770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.104826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.104868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.105020 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.105081 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.105064254 +0000 UTC m=+39.397933972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.105129 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.105158 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.105150871 +0000 UTC m=+39.398020585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.105195 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:55.105267 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:55.105237 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.105229464 +0000 UTC m=+39.398099177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:55.411255 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.411211 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cjhm7" event={"ID":"3d6a461d-3472-4285-b5a9-1a11367f6497","Type":"ContainerStarted","Data":"45c4d1b188d1cdd7b4d687f79bb09a7ae5ce17ffa7a2876f210908661b081cf8"} Nov 04 03:16:55.417056 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.416960 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" event={"ID":"3194fa09-26b5-46ff-bada-c1077f024432","Type":"ContainerStarted","Data":"278b44c9dfa51e22a343986e9ed58af4c71f10496cb3a1ac2cc64b050ead4730"} Nov 04 03:16:55.421154 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.421069 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gxvcp" event={"ID":"969e817f-6f19-4c15-835c-abcb599d5554","Type":"ContainerStarted","Data":"76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f"} Nov 04 03:16:55.533882 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.532164 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" podStartSLOduration=367.71333513 podStartE2EDuration="6m10.532147563s" totalImagesPullingTime="2.818812433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:55.531926913 +0000 UTC m=+35.824796648" watchObservedRunningTime="2026-11-04 03:16:55.532147563 +0000 UTC m=+35.825017300" Nov 04 03:16:55.755216 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:55.755133 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:56.006068 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:56.005985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:57.433564 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:57.432813 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/0.log" Nov 04 03:16:57.433564 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:57.432845 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ed988d6-f52e-4030-8805-1c214876ee88" containerID="ba54a0855a003ee5b386f6cb13e9c41e7ce44c67655c333f31616c07dce6519b" exitCode=255 Nov 04 03:16:57.433564 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:57.432904 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" event={"ID":"0ed988d6-f52e-4030-8805-1c214876ee88","Type":"ContainerDied","Data":"ba54a0855a003ee5b386f6cb13e9c41e7ce44c67655c333f31616c07dce6519b"} Nov 04 03:16:57.433564 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:57.433148 2579 scope.go:122] "RemoveContainer" containerID="ba54a0855a003ee5b386f6cb13e9c41e7ce44c67655c333f31616c07dce6519b" Nov 04 03:16:57.445749 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:57.445585 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" event={"ID":"d0cabe74-608b-461e-a3aa-fa049a28de5b","Type":"ContainerStarted","Data":"1a398c2fece9be60931f30f76910cf2d67c67e073550bfb5af3796cb571d57aa"} Nov 04 03:16:57.670615 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:57.670538 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" podStartSLOduration=368.412511286 podStartE2EDuration="6m13.670523303s" totalImagesPullingTime="5.258012017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:57.669026579 +0000 UTC m=+37.961896316" watchObservedRunningTime="2026-11-04 03:16:57.670523303 +0000 UTC m=+37.963393038" Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.452218 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" event={"ID":"63620800-28b7-4d31-88d8-b75f34a5dda1","Type":"ContainerStarted","Data":"b938883b564ddd4b19a452be62e8a8fc9043a41b8a46e5a2650e8b1cb99bb68a"} Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.454242 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/1.log" Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.454790 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/0.log" Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.454822 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ed988d6-f52e-4030-8805-1c214876ee88" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" exitCode=255 Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.454875 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" event={"ID":"0ed988d6-f52e-4030-8805-1c214876ee88","Type":"ContainerDied","Data":"6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80"} Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.454913 2579 scope.go:122] "RemoveContainer" containerID="ba54a0855a003ee5b386f6cb13e9c41e7ce44c67655c333f31616c07dce6519b" Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.455213 2579 scope.go:122] "RemoveContainer" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:58.455365 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5cxt_openshift-console-operator(0ed988d6-f52e-4030-8805-1c214876ee88)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" Nov 04 03:16:58.462523 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.458610 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" event={"ID":"86941193-b41d-47a0-b4cf-ef4221272e45","Type":"ContainerStarted","Data":"a19f86492e0203f4e7221dc4abbbeb6e55c4991e48ef9f641cf60e4ed6b1be4f"} Nov 04 03:16:58.569873 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.569816 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" podStartSLOduration=372.004898128 podStartE2EDuration="6m17.569797192s" totalImagesPullingTime="5.564899064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:58.568619262 +0000 UTC m=+38.861488997" watchObservedRunningTime="2026-11-04 03:16:58.569797192 +0000 UTC m=+38.862666924" Nov 04 03:16:58.687291 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.687227 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" podStartSLOduration=366.858860789 podStartE2EDuration="6m12.687206825s" totalImagesPullingTime="5.828346036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:58.685633903 +0000 UTC m=+38.978503632" watchObservedRunningTime="2026-11-04 03:16:58.687206825 +0000 UTC m=+38.980076561" Nov 04 03:16:58.812485 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.812438 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c44w7"] Nov 04 03:16:58.846075 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.846047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:58.874587 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.874080 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c44w7"] Nov 04 03:16:58.898877 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.898849 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 03:16:58.899049 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.898937 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 03:16:58.899113 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.899061 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmpr4\"" Nov 04 03:16:58.950931 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.950897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24761c51-4ac8-4976-b04f-91f529dd055e-data-volume\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:58.951092 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.950968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:58.951092 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.950994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzvl\" (UniqueName: \"kubernetes.io/projected/24761c51-4ac8-4976-b04f-91f529dd055e-kube-api-access-9tzvl\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:58.951092 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.951038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24761c51-4ac8-4976-b04f-91f529dd055e-crio-socket\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:58.951092 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:58.951072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24761c51-4ac8-4976-b04f-91f529dd055e-data-volume\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.052355 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.052416 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.052399866 +0000 UTC m=+47.345269593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.052539 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.052553 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052784 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24761c51-4ac8-4976-b04f-91f529dd055e-data-volume\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.052841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.052826273 +0000 UTC m=+47.345695991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzvl\" (UniqueName: \"kubernetes.io/projected/24761c51-4ac8-4976-b04f-91f529dd055e-kube-api-access-9tzvl\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24761c51-4ac8-4976-b04f-91f529dd055e-crio-socket\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.052995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.053376 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053099 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053123 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053142 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.553128149 +0000 UTC m=+39.845997875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053158 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:16:59.553147873 +0000 UTC m=+39.846017588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : configmap "kube-rbac-proxy" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053174 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.053164792 +0000 UTC m=+47.346034505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053204 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053215 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053227 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053245 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.053234806 +0000 UTC m=+47.346104533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.053260 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.053250776 +0000 UTC m=+47.346120493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:16:59.054263 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.053302 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24761c51-4ac8-4976-b04f-91f529dd055e-crio-socket\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.122251 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.122209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzvl\" (UniqueName: \"kubernetes.io/projected/24761c51-4ac8-4976-b04f-91f529dd055e-kube-api-access-9tzvl\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.153687 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.153657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:16:59.153860 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.153696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:16:59.153860 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.153738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:16:59.153860 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.153807 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:16:59.154030 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.153872 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.153852794 +0000 UTC m=+47.446722528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:16:59.154030 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.153876 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:16:59.154362 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.154340 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.154314665 +0000 UTC m=+47.447184394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:16:59.159503 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.159373 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:59.159503 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.159456 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.159439036 +0000 UTC m=+47.452308753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:16:59.214702 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.214659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:59.464754 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.464720 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/1.log" Nov 04 03:16:59.465204 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.465161 2579 scope.go:122] "RemoveContainer" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" Nov 04 03:16:59.465533 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.465347 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5cxt_openshift-console-operator(0ed988d6-f52e-4030-8805-1c214876ee88)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" Nov 04 03:16:59.467961 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.467723 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zh4vg" event={"ID":"056b218a-8def-4ac5-8a9b-78b69b08943d","Type":"ContainerStarted","Data":"c4f60731ad633519f5046646f37885fe1d2a5061d5eb8ddca35f3e83b66f8281"} Nov 04 03:16:59.469079 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.469017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:59.559171 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.559135 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:16:59.559327 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.559221 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:00.559198962 +0000 UTC m=+40.852068698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : configmap "kube-rbac-proxy" not found Nov 04 03:16:59.559532 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.558765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.559655 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.559623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:16:59.560442 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.560423 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:16:59.560570 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:16:59.560498 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:00.560462587 +0000 UTC m=+40.853332317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:16:59.727988 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.727894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:16:59.858203 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.858135 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zh4vg" podStartSLOduration=2.448836849 podStartE2EDuration="8.858113731s" totalImagesPullingTime="6.409276882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:16:59.708302456 +0000 UTC m=+40.001172191" watchObservedRunningTime="2026-11-04 03:16:59.858113731 +0000 UTC m=+40.150983466" Nov 04 03:16:59.859286 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.859259 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74"] Nov 04 03:16:59.877293 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.877256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:16:59.920849 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.920792 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74"] Nov 04 03:16:59.930240 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.930188 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 03:16:59.930240 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.930188 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 03:16:59.930446 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.930276 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sg2zz\"" Nov 04 03:16:59.963301 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.963268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxd5\" (UniqueName: \"kubernetes.io/projected/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1-kube-api-access-sdxd5\") pod \"migrator-66df5d496b-8sv74\" (UID: \"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:16:59.977650 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:16:59.977618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:00.064032 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.063945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxd5\" (UniqueName: \"kubernetes.io/projected/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1-kube-api-access-sdxd5\") pod \"migrator-66df5d496b-8sv74\" (UID: \"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:17:00.125997 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.125931 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxd5\" (UniqueName: \"kubernetes.io/projected/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1-kube-api-access-sdxd5\") pod \"migrator-66df5d496b-8sv74\" (UID: \"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:17:00.207979 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.207741 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:17:00.224722 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.224662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:00.474079 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.474045 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" event={"ID":"939eba33-05f1-4a74-97e4-3c41074ca9ac","Type":"ContainerStarted","Data":"a35d43348a018084f6797e224ea06dec360043e94c67aff346a87130c0ca4e7d"} Nov 04 03:17:00.576012 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.575974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:00.576198 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:17:00.576057 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 04 03:17:00.576198 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.576084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:00.576320 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:17:00.576206 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:02.576180766 +0000 UTC m=+42.869050484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : configmap "kube-rbac-proxy" not found Nov 04 03:17:00.576320 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:17:00.576223 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:17:00.576320 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:17:00.576271 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:02.576255857 +0000 UTC m=+42.869125584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:17:00.591310 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.590769 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" podStartSLOduration=362.321338283 podStartE2EDuration="6m9.590755464s" totalImagesPullingTime="7.269417181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:00.59071602 +0000 UTC m=+40.883585757" watchObservedRunningTime="2026-11-04 03:17:00.590755464 +0000 UTC m=+40.883625199" Nov 04 03:17:00.620737 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.620692 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74"] Nov 04 03:17:00.634370 ip-10-0-132-184 kubenswrapper[2579]: W1104 03:17:00.634337 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fe2170_93d8_4926_b4c3_3d0bdfb6abe1.slice/crio-97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5 WatchSource:0}: Error finding container 97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5: Status 404 returned error can't find the container with id 97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5 Nov 04 03:17:00.928015 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:00.927974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:01.197345 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.197265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:01.460066 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.459984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:01.479718 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.479677 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" event={"ID":"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1","Type":"ContainerStarted","Data":"97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5"} Nov 04 03:17:01.481526 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.481500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gxvcp" event={"ID":"969e817f-6f19-4c15-835c-abcb599d5554","Type":"ContainerStarted","Data":"35cb462ef4e074b59109888cbc642449ae5702e40f03159bf2658dc91c01715d"} Nov 04 03:17:01.584787 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.584745 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:01.584945 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.584803 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:01.585200 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.585183 2579 scope.go:122] "RemoveContainer" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" Nov 04 03:17:01.585387 ip-10-0-132-184 kubenswrapper[2579]: E1104 03:17:01.585368 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5cxt_openshift-console-operator(0ed988d6-f52e-4030-8805-1c214876ee88)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" Nov 04 03:17:01.600234 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:01.600177 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gxvcp" podStartSLOduration=35.798973684 podStartE2EDuration="41.600165339s" totalImagesPullingTime="5.801191655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:01.598680621 +0000 UTC m=+41.891550358" watchObservedRunningTime="2026-11-04 03:17:01.600165339 +0000 UTC m=+41.893035073" Nov 04 03:17:02.241131 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:02.241104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:02.486670 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:02.486582 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cjhm7" event={"ID":"3d6a461d-3472-4285-b5a9-1a11367f6497","Type":"ContainerStarted","Data":"c740448a2055bd0ff0bfb4784eab80b9a2c71c6be26b8a07dd38e6ccefe870d7"} Nov 04 03:17:02.487041 ip-10-0-132-184 kubenswrapper[2579]: I1104 03:17:02.486698 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:17:02.540259 ip-10-0-132-184 systemd[1]: Stopping Kubernetes Kubelet... Nov 04 03:17:02.551860 ip-10-0-132-184 systemd[1]: kubelet.service: Deactivated successfully. Nov 04 03:17:02.552025 ip-10-0-132-184 systemd[1]: Stopped Kubernetes Kubelet. Nov 04 03:17:02.552073 ip-10-0-132-184 systemd[1]: kubelet.service: Consumed 3.154s CPU time. Nov 04 03:17:02.562203 ip-10-0-132-184 systemd[1]: Starting Kubernetes Kubelet... Nov 04 03:17:02.650095 ip-10-0-132-184 kubenswrapper[4959]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:17:02.650095 ip-10-0-132-184 kubenswrapper[4959]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 04 03:17:02.650095 ip-10-0-132-184 kubenswrapper[4959]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:17:02.650095 ip-10-0-132-184 kubenswrapper[4959]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 04 03:17:02.652195 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652097 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.652195 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652192 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.652195 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652196 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.652195 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652199 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.652195 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652201 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.652195 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652204 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652207 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652210 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652213 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652215 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652218 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652222 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652227 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652230 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652233 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652236 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652239 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652241 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652244 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652247 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652249 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652252 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652255 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652258 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652260 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.652419 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652263 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652266 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652268 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652271 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652273 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652276 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652279 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652282 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652284 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652287 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652289 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652292 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652295 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652297 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652300 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652302 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652305 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652307 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652310 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.652948 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652313 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652315 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652318 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652320 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652323 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652326 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652329 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652331 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652334 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652336 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652339 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652341 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652344 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652346 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652348 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652353 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652357 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652360 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652363 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.653411 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652366 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652368 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652371 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652373 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652376 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652378 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652381 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652384 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652386 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652389 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652391 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652394 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652396 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652399 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652401 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652404 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652406 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652410 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652412 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652415 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.653887 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652417 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652420 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652422 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652425 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652428 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652430 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652433 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652435 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652438 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652441 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652443 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652446 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652449 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652452 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652454 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652457 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652459 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652462 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652480 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.654371 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652483 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652486 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652489 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.652492 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653273 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653281 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653284 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653286 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653289 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653292 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653295 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653298 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653301 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653303 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653306 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653308 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653311 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653313 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653316 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.654889 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653319 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653322 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653325 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653327 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653330 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653335 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653338 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653341 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653343 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653346 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653348 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653351 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653354 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653356 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653359 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653361 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653364 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653367 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653370 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653373 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.655354 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653375 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653379 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653382 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653385 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653387 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653390 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653392 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653395 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653397 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653400 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653402 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653405 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653407 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653409 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653413 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653416 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653419 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653422 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653425 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653428 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.656176 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653430 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653432 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653436 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653441 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653444 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653446 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653449 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653451 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653454 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653456 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653459 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653462 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653480 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653484 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653486 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653489 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653491 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653494 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653497 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.656996 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653499 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653502 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653504 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653506 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653509 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653511 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653514 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653517 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653519 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653522 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653525 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653528 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653530 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653534 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653537 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653539 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653542 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653544 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653547 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653549 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.657452 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653552 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653554 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653557 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653560 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653563 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653566 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653569 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653571 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653575 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653578 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653581 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.653583 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654120 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654125 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654128 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654131 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654134 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654136 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654139 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654142 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.658038 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654145 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654147 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654152 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654154 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654157 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654159 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654162 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654164 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654167 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654169 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654172 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654174 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654178 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654183 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654186 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654189 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654192 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654194 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654197 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.658540 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654199 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654202 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654204 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654207 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654209 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654212 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654214 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654217 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654219 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654221 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654224 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654226 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654229 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654231 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654234 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654238 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654240 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654243 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654245 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654248 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.659017 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654250 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654253 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654255 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654257 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654260 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654262 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654266 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654268 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654271 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654273 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654276 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654279 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654281 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654283 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654286 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654289 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654291 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654293 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654296 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654298 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.659646 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654301 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654304 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654306 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654309 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654311 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654314 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654316 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654320 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654323 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654325 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654328 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654330 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654335 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654338 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654341 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654343 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654346 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654348 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654351 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654354 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.660295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654356 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654359 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654361 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654364 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654367 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654369 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654372 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654375 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654377 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654380 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654383 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654385 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654389 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654392 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654395 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654397 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654400 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654402 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.654405 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.660903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654523 4959 flags.go:64] FLAG: --address="0.0.0.0" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654533 4959 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654541 4959 flags.go:64] FLAG: --anonymous-auth="true" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654546 4959 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654551 4959 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654554 4959 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654558 4959 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654563 4959 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654566 4959 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654570 4959 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654574 4959 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654577 4959 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654580 4959 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654583 4959 flags.go:64] FLAG: --cgroup-root="" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654586 4959 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654589 4959 flags.go:64] FLAG: --client-ca-file="" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654591 4959 flags.go:64] FLAG: --cloud-provider="external" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654595 4959 flags.go:64] FLAG: --cluster-dns="[]" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654600 4959 flags.go:64] FLAG: --cluster-domain="" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654602 4959 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654605 4959 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654609 4959 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654612 4959 flags.go:64] FLAG: --container-log-max-files="5" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654616 4959 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 04 03:17:02.661403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654619 4959 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654622 4959 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654625 4959 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654629 4959 flags.go:64] FLAG: --contention-profiling="false" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654632 4959 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654635 4959 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654638 4959 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654640 4959 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654645 4959 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654648 4959 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654652 4959 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654655 4959 flags.go:64] FLAG: --enable-load-reader="false" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654658 4959 flags.go:64] FLAG: --enable-server="true" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654661 4959 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654666 4959 flags.go:64] FLAG: --event-burst="100" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654671 4959 flags.go:64] FLAG: --event-qps="50" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654674 4959 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654677 4959 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654680 4959 flags.go:64] FLAG: --eviction-hard="" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654684 4959 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654688 4959 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654691 4959 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654694 4959 flags.go:64] FLAG: --eviction-soft="" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654698 4959 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654703 4959 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 04 03:17:02.662116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654708 4959 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654713 4959 flags.go:64] FLAG: --experimental-mounter-path="" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654718 4959 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654722 4959 flags.go:64] FLAG: --fail-swap-on="true" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654727 4959 flags.go:64] FLAG: --feature-gates="" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654733 4959 flags.go:64] FLAG: --file-check-frequency="20s" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654737 4959 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654740 4959 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654744 4959 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654747 4959 flags.go:64] FLAG: --healthz-port="10248" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654750 4959 flags.go:64] FLAG: --help="false" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654753 4959 flags.go:64] FLAG: --hostname-override="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654756 4959 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654759 4959 flags.go:64] FLAG: --http-check-frequency="20s" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654762 4959 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654765 4959 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654768 4959 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654771 4959 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654774 4959 flags.go:64] FLAG: --image-service-endpoint="" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654777 4959 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654780 4959 flags.go:64] FLAG: --kube-api-burst="100" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654783 4959 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654787 4959 flags.go:64] FLAG: --kube-api-qps="50" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654790 4959 flags.go:64] FLAG: --kube-reserved="" Nov 04 03:17:02.662836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654793 4959 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654796 4959 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654799 4959 flags.go:64] FLAG: --kubelet-cgroups="" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654801 4959 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654804 4959 flags.go:64] FLAG: --lock-file="" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654808 4959 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654810 4959 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654813 4959 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654819 4959 flags.go:64] FLAG: --log-json-split-stream="false" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654822 4959 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654826 4959 flags.go:64] FLAG: --log-text-split-stream="false" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654828 4959 flags.go:64] FLAG: --logging-format="text" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654831 4959 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654835 4959 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654837 4959 flags.go:64] FLAG: --manifest-url="" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654840 4959 flags.go:64] FLAG: --manifest-url-header="" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654845 4959 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654848 4959 flags.go:64] FLAG: --max-open-files="1000000" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654874 4959 flags.go:64] FLAG: --max-pods="110" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654878 4959 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654882 4959 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654885 4959 flags.go:64] FLAG: --memory-manager-policy="None" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654889 4959 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654892 4959 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654895 4959 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 04 03:17:02.663410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654898 4959 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654906 4959 flags.go:64] FLAG: --node-status-max-images="50" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654909 4959 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654912 4959 flags.go:64] FLAG: --oom-score-adj="-999" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654915 4959 flags.go:64] FLAG: --pod-cidr="" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654923 4959 flags.go:64] FLAG: --pod-manifest-path="" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654926 4959 flags.go:64] FLAG: --pod-max-pids="-1" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654929 4959 flags.go:64] FLAG: --pods-per-core="0" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654932 4959 flags.go:64] FLAG: --port="10250" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654935 4959 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654938 4959 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0d5a0a98334663306" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654941 4959 flags.go:64] FLAG: --qos-reserved="" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654944 4959 flags.go:64] FLAG: --read-only-port="10255" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654947 4959 flags.go:64] FLAG: --register-node="true" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654949 4959 flags.go:64] FLAG: --register-with-taints="" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654953 4959 flags.go:64] FLAG: --registry-burst="10" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654956 4959 flags.go:64] FLAG: --registry-qps="5" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654959 4959 flags.go:64] FLAG: --reserved-cpus="" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654964 4959 flags.go:64] FLAG: --reserved-memory="" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654968 4959 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654971 4959 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654974 4959 flags.go:64] FLAG: --rotate-certificates="false" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654977 4959 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654980 4959 flags.go:64] FLAG: --runonce="false" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654983 4959 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654986 4959 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654989 4959 flags.go:64] FLAG: --seccomp-default="false" Nov 04 03:17:02.664043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654992 4959 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654994 4959 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.654997 4959 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655001 4959 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655004 4959 flags.go:64] FLAG: --storage-driver-password="root" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655007 4959 flags.go:64] FLAG: --storage-driver-secure="false" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655010 4959 flags.go:64] FLAG: --storage-driver-table="stats" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655013 4959 flags.go:64] FLAG: --storage-driver-user="root" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655016 4959 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655019 4959 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655023 4959 flags.go:64] FLAG: --system-cgroups="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655026 4959 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655031 4959 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655034 4959 flags.go:64] FLAG: --tls-cert-file="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655037 4959 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655041 4959 flags.go:64] FLAG: --tls-min-version="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655044 4959 flags.go:64] FLAG: --tls-private-key-file="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655047 4959 flags.go:64] FLAG: --topology-manager-policy="none" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655050 4959 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655053 4959 flags.go:64] FLAG: --topology-manager-scope="container" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655056 4959 flags.go:64] FLAG: --v="2" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655060 4959 flags.go:64] FLAG: --version="false" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655065 4959 flags.go:64] FLAG: --vmodule="" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655070 4959 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655075 4959 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 04 03:17:02.664716 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655220 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655224 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655227 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655230 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655232 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655235 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655237 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655240 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655242 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655245 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655248 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655250 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655252 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655255 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655258 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655260 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655264 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655269 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655272 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655275 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.665317 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655278 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655280 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655283 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655285 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655288 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655290 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655293 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655295 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655297 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655300 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655306 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655309 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655313 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655316 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655318 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655321 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655324 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655326 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655329 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.665833 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655331 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655334 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655337 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655339 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655341 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655344 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655346 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655349 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655351 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655353 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655357 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655360 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655362 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655365 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655368 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655371 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655373 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655375 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655378 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655380 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.666295 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655383 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655385 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655388 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655392 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655395 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655398 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655401 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655404 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655406 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655408 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655411 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655413 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655416 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655418 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655421 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655423 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655426 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655428 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655431 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655433 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.666938 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655436 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655438 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655441 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655444 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655447 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655449 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655451 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655454 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655456 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655459 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655462 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655480 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655484 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655488 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655492 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655496 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655499 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655501 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655504 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.667597 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655507 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655510 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655512 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655515 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655517 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655520 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655522 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655525 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.655555 4959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655590 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655595 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655599 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655602 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655605 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655607 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.668279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655611 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655614 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655616 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655619 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655622 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655624 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655627 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655629 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655632 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655634 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655637 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655639 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655642 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655644 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655650 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655653 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655655 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655658 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655661 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655664 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.668858 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655666 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655668 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655671 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655674 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655676 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655678 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655681 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655683 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655686 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655688 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655690 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655693 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655696 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655699 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655701 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655704 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655706 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655709 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655711 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.669344 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655723 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655726 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655728 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655731 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655734 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655737 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655739 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655742 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655745 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655748 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655750 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655753 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655755 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655758 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655761 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655763 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655766 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655768 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655771 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655773 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.669947 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655776 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655781 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655784 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655789 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655793 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655799 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655802 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655805 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655809 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655814 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655818 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655825 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655832 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655836 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655839 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655842 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655845 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655847 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655851 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655856 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.670544 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655859 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655861 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655864 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655866 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.655869 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656232 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656238 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656240 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656244 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656246 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656249 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656253 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656256 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656259 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656261 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656264 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656267 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656271 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656273 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.671102 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656276 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.656278 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.661873 4959 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.661901 4959 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.661920 4959 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.661925 4959 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.661986 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.661993 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.661997 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662000 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662003 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662005 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662008 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662010 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662013 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662016 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662018 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662021 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662024 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662026 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.671596 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662029 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662032 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662034 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662038 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662040 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662043 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662046 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662048 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662051 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662060 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662062 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662065 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662068 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662070 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662073 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662075 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662078 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662080 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662083 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662086 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.672099 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662088 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662090 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662093 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662096 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662098 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662101 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662104 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662106 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662110 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662112 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662115 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662117 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662121 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662125 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662131 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662136 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662140 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662144 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662148 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662153 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.672599 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662163 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662167 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662170 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662172 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662175 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662177 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662180 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662182 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662185 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662187 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662190 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662193 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662195 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662198 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662200 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662202 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662205 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662208 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662211 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662213 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.673077 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662216 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662219 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662221 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662224 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662226 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662229 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662232 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662234 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662237 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662239 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662241 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662244 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662247 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662249 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662252 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662254 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662256 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662259 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662261 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.673647 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662264 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662266 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662269 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662273 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662277 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662280 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662282 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662285 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662289 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662293 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662297 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662302 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662306 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.662352 4959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662557 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.674165 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662567 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662573 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662577 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662582 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662588 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662593 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662597 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662602 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662606 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662610 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662615 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662620 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662624 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662628 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662632 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662637 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662641 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662645 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662649 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662654 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.674695 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662658 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662662 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662666 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662670 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662674 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662678 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662710 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662717 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662723 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662728 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662733 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662737 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662741 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662745 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662750 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662754 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662758 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662763 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662767 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662771 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.675436 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662775 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662779 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662784 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662788 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662792 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662796 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662801 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662805 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662809 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662813 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662817 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662822 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662826 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662830 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662834 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662839 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662844 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662848 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662852 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662856 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.676207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662860 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662865 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662869 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662873 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662877 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662881 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662885 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662890 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662894 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662898 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662903 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662907 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662912 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662916 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662920 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662924 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662928 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662932 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662936 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.676903 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662941 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662944 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662949 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662953 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662957 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662961 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662965 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662969 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662976 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662982 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662987 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662991 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.662996 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663000 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663007 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663011 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663015 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663019 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663023 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.677410 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663027 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663032 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663036 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663040 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663045 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663050 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663055 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.663094 4959 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663148 4959 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663152 4959 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663155 4959 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663158 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663160 4959 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663163 4959 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663165 4959 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 04 03:17:02.677906 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663168 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663170 4959 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663173 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663175 4959 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663178 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663180 4959 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663183 4959 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663186 4959 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663188 4959 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663191 4959 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663193 4959 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663196 4959 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663199 4959 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663203 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663205 4959 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663208 4959 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663210 4959 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663213 4959 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663216 4959 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 04 03:17:02.678279 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663221 4959 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663224 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663228 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663230 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663234 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663236 4959 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663239 4959 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663241 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663244 4959 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663246 4959 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663249 4959 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663251 4959 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663254 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663256 4959 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663258 4959 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663261 4959 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663264 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663268 4959 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663271 4959 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 04 03:17:02.678756 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663274 4959 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663276 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663279 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663281 4959 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663284 4959 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663286 4959 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663289 4959 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663293 4959 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663295 4959 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663297 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663300 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663302 4959 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663305 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663307 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663310 4959 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663313 4959 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663316 4959 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663318 4959 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663321 4959 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663324 4959 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 04 03:17:02.679294 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663327 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663329 4959 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663332 4959 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663334 4959 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663337 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663339 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663342 4959 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663345 4959 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663347 4959 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663350 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663353 4959 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663355 4959 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663357 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663360 4959 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663362 4959 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663365 4959 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663367 4959 feature_gate.go:386] unrecognized feature gate: EVPN Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663370 4959 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663372 4959 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663376 4959 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 04 03:17:02.679830 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663379 4959 feature_gate.go:386] unrecognized feature gate: Example2 Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663381 4959 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663383 4959 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663386 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663388 4959 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663391 4959 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663394 4959 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663398 4959 feature_gate.go:386] unrecognized feature gate: Example Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663400 4959 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663403 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663405 4959 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663408 4959 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663411 4959 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663414 4959 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663417 4959 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663420 4959 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663423 4959 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663425 4959 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663428 4959 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663430 4959 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 04 03:17:02.680333 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:02.663433 4959 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.663514 4959 server.go:957] "Client rotation is on, will bootstrap in background" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.664541 4959 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.664603 4959 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.664822 4959 server.go:1014] "Starting client certificate rotation" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.664958 4959 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.665002 4959 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-03 03:11:20 +0000 UTC" deadline="2028-08-13 09:30:01.463694692 +0000 UTC" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.665030 4959 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15558h12m58.798666963s" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.665337 4959 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.666606 4959 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.667927 4959 log.go:25] "Validated CRI v1 runtime API" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.669591 4959 log.go:25] "Validated CRI v1 image API" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.670280 4959 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 04 03:17:02.680876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.672162 4959 fs.go:135] Filesystem UUIDs: map[71cacd08-7b0a-4ba2-8e1e-d67a070fd986:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 89e34c74-3900-46a4-acb2-f8bed21ec0ac:/dev/nvme0n1p3] Nov 04 03:17:02.681225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.672181 4959 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45c4d1b188d1cdd7b4d687f79bb09a7ae5ce17ffa7a2876f210908661b081cf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45c4d1b188d1cdd7b4d687f79bb09a7ae5ce17ffa7a2876f210908661b081cf8/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/617a4896bb5a543bff4eb8a49ee5db024c7be3e8df5c290171e89a0d532e7f20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/617a4896bb5a543bff4eb8a49ee5db024c7be3e8df5c290171e89a0d532e7f20/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68892cba95d574b1321399d832b55986a6b2c7b1735a676da4ae55c4a5987ab9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68892cba95d574b1321399d832b55986a6b2c7b1735a676da4ae55c4a5987ab9/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94d498c93fdaba6a346e7e2eeda97dfc6acee5b5265f36f2dafa182a4d82db80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94d498c93fdaba6a346e7e2eeda97dfc6acee5b5265f36f2dafa182a4d82db80/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0d2c34775ea931e4830a5f5b7667032aad241bd0cf5fa9ae068dd72b869874a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0d2c34775ea931e4830a5f5b7667032aad241bd0cf5fa9ae068dd72b869874a/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c140d95316b4967461aa10a79a2e19041d9403334de8fc559103d94a68384fd5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c140d95316b4967461aa10a79a2e19041d9403334de8fc559103d94a68384fd5/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6f7f6313aa02a38a5ee7c17ebc47d1e5644e4e070e8b9b576e21574eca18c82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6f7f6313aa02a38a5ee7c17ebc47d1e5644e4e070e8b9b576e21574eca18c82/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b218a-8def-4ac5-8a9b-78b69b08943d/volumes/kubernetes.io~projected/kube-api-access-m4clk:{mountpoint:/var/lib/kubelet/pods/056b218a-8def-4ac5-8a9b-78b69b08943d/volumes/kubernetes.io~projected/kube-api-access-m4clk major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056b218a-8def-4ac5-8a9b-78b69b08943d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/056b218a-8def-4ac5-8a9b-78b69b08943d/volumes/kubernetes.io~secret/cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074a2389-db4e-4779-979e-0b9ba07a65bf/volumes/kubernetes.io~projected/kube-api-access-lzppv:{mountpoint:/var/lib/kubelet/pods/074a2389-db4e-4779-979e-0b9ba07a65bf/volumes/kubernetes.io~projected/kube-api-access-lzppv major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ed988d6-f52e-4030-8805-1c214876ee88/volumes/kubernetes.io~projected/kube-api-access-pf7dg:{mountpoint:/var/lib/kubelet/pods/0ed988d6-f52e-4030-8805-1c214876ee88/volumes/kubernetes.io~projected/kube-api-access-pf7dg major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ed988d6-f52e-4030-8805-1c214876ee88/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0ed988d6-f52e-4030-8805-1c214876ee88/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24761c51-4ac8-4976-b04f-91f529dd055e/volumes/kubernetes.io~projected/kube-api-access-9tzvl:{mountpoint:/var/lib/kubelet/pods/24761c51-4ac8-4976-b04f-91f529dd055e/volumes/kubernetes.io~projected/kube-api-access-9tzvl major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/284dd9ee-7ea1-4c8d-9674-57c15be65836/volumes/kubernetes.io~projected/kube-api-access-c7mwd:{mountpoint:/var/lib/kubelet/pods/284dd9ee-7ea1-4c8d-9674-57c15be65836/volumes/kubernetes.io~projected/kube-api-access-c7mwd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3194fa09-26b5-46ff-bada-c1077f024432/volumes/kubernetes.io~projected/kube-api-access-2979t:{mountpoint:/var/lib/kubelet/pods/3194fa09-26b5-46ff-bada-c1077f024432/volumes/kubernetes.io~projected/kube-api-access-2979t major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d6a461d-3472-4285-b5a9-1a11367f6497/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3d6a461d-3472-4285-b5a9-1a11367f6497/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9891bc-c81c-4ce5-afc9-75c07b82176a/volumes/kubernetes.io~projected/kube-api-access-s2x88:{mountpoint:/var/lib/kubelet/pods/3e9891bc-c81c-4ce5-afc9-75c07b82176a/volumes/kubernetes.io~projected/kube-api-access-s2x88 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63620800-28b7-4d31-88d8-b75f34a5dda1/volumes/kubernetes.io~projected/kube-api-access-htnz4:{mountpoint:/var/lib/kubelet/pods/63620800-28b7-4d31-88d8-b75f34a5dda1/volumes/kubernetes.io~projected/kube-api-access-htnz4 major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63620800-28b7-4d31-88d8-b75f34a5dda1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/63620800-28b7-4d31-88d8-b75f34a5dda1/volumes/kubernetes.io~secret/serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~projected/kube-api-access-srm9p:{mountpoint:/var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~projected/kube-api-access-srm9p major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/781b84f6-581f-4357-b447-8d4f92aece81/volumes/kubernetes.io~projected/kube-api-access-4l5sg:{mountpoint:/var/lib/kubelet/pods/781b84f6-581f-4357-b447-8d4f92aece81/volumes/kubernetes.io~projected/kube-api-access-4l5sg major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~projected/kube-api-access-9t6hr:{mountpoint:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~projected/kube-api-access-9t6hr major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86941193-b41d-47a0-b4cf-ef4221272e45/volumes/kubernetes.io~projected/kube-api-access-rc2hs:{mountpoint:/var/lib/kubelet/pods/86941193-b41d-47a0-b4cf-ef4221272e45/volumes/kubernetes.io~projected/kube-api-access-rc2hs major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86941193-b41d-47a0-b4cf-ef4221272e45/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/86941193-b41d-47a0-b4cf-ef4221272e45/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9380c359-acb2-43f6-af23-89baa5bee3b4/volumes/kubernetes.io~projected/kube-api-access-fb2r9:{mountpoint:/var/lib/kubelet/pods/9380c359-acb2-43f6-af23-89baa5bee3b4/volumes/kubernetes.io~projected/kube-api-access-fb2r9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/939eba33-05f1-4a74-97e4-3c41074ca9ac/volumes/kubernetes.io~projected/kube-api-access-t5z2j:{mountpoint:/var/lib/kubelet/pods/939eba33-05f1-4a74-97e4-3c41074ca9ac/volumes/kubernetes.io~projected/kube-api-access-t5z2j major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/969e817f-6f19-4c15-835c-abcb599d5554/volumes/kubernetes.io~projected/kube-api-access-vl7nn:{mountpoint:/var/lib/kubelet/pods/969e817f-6f19-4c15-835c-abcb599d5554/volumes/kubernetes.io~projected/kube-api-access-vl7nn major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/986942d0-73ec-4567-9026-48482bd186ae/volumes/kubernetes.io~projected/kube-api-access-j2z99:{mountpoint:/var/lib/kubelet/pods/986942d0-73ec-4567-9026-48482bd186ae/volumes/kubernetes.io~projected/kube-api-access-j2z99 major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a699a906-fae8-47d9-92ba-8eefce6cf724/volumes/kubernetes.io~projected/kube-api-access-74fmz:{mountpoint:/var/lib/kubelet/pods/a699a906-fae8-47d9-92ba-8eefce6cf724/volumes/kubernetes.io~projected/kube-api-access-74fmz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~projected/kube-api-access-gcz2w:{mountpoint:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~projected/kube-api-access-gcz2w major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad24bd15-2732-4e05-9494-ca0691fd9e86/volumes/kubernetes.io~projected/kube-api-access-487vl:{mountpoint:/var/lib/kubelet/pods/ad24bd15-2732-4e05-9494-ca0691fd9e86/volumes/kubernetes.io~projected/kube-api-access-487vl major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~projected/kube-api-access-gx8hb:{mountpoint:/var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~projected/kube-api-access-gx8hb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48/volumes/kubernetes.io~projected/kube-api-access-tkhwc:{mountpoint:/var/lib/kubelet/pods/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48/volumes/kubernetes.io~projected/kube-api-access-tkhwc major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1/volumes/kubernetes.io~projected/kube-api-access-sdxd5:{mountpoint:/var/lib/kubelet/pods/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1/volumes/kubernetes.io~projected/kube-api-access-sdxd5 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volumes/kubernetes.io~projected/kube-api-access-sd27k:{mountpoint:/var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volumes/kubernetes.io~projected/kube-api-access-sd27k major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0cabe74-608b-461e-a3aa-fa049a28de5b/volumes/kubernetes.io~projected/kube-api-access-vnts4:{mountpoint:/var/lib/kubelet/pods/d0cabe74-608b-461e-a3aa-fa049a28de5b/volumes/kubernetes.io~projected/kube-api-access-vnts4 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0cabe74-608b-461e-a3aa-fa049a28de5b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d0cabe74-608b-461e-a3aa-fa049a28de5b/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4132467-8852-45ae-8c94-bf0d78847bc6/volumes/kubernetes.io~projected/kube-api-access-lns92:{mountpoint:/var/lib/kubelet/pods/d4132467-8852-45ae-8c94-bf0d78847bc6/volumes/kubernetes.io~projected/kube-api-access-lns92 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab481c8-cff5-4c29-aa52-552434b043bf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dab481c8-cff5-4c29-aa52-552434b043bf/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e04fdd96-e8ac-4338-933c-1b9dadb56b89/volumes/kubernetes.io~projected/kube-api-access-kq9pz:{mountpoint:/var/lib/kubelet/pods/e04fdd96-e8ac-4338-933c-1b9dadb56b89/volumes/kubernetes.io~projected/kube-api-access-kq9pz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4b5da75-9530-4e0b-849d-fa50fdec3241/volumes/kubernetes.io~projected/kube-api-access-kcsqq:{mountpoint:/var/lib/kubelet/pods/e4b5da75-9530-4e0b-849d-fa50fdec3241/volumes/kubernetes.io~projected/kube-api-access-kcsqq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f762673f-00d6-4182-b7e3-bd16847fd218/volumes/kubernetes.io~projected/kube-api-access-wgkz8:{mountpoint:/var/lib/kubelet/pods/f762673f-00d6-4182-b7e3-bd16847fd218/volumes/kubernetes.io~projected/kube-api-access-wgkz8 major:0 minor:209 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/79173cfa8636d8b7cf317d2aa16d2691f2ebc8b542324c4769f942806ac965d8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a24548c3271b682a9f3dc741b291223772896d55d1c20efdb9848d26b1d40899/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2d89960d6bc181427e2ff6086596fd227dadb1c8dafcb51ea232b0c8e2715695/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0e95d10359307f63739c36f31a9fefe69301cac26f470fde45aa12234b0e71e7/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/19cb5fa28a453bb48478921223379472cdd3985e93bdc5b8e2d9278c98166cef/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6d089ad317797aef1e857e0cca60e1f0db62230494437a62b1ee9adf5cc9e47a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/289ed8b664b00d2a34d55d408a3b85de7d25cf6953c24800eb935988d5e6341d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/09b4603aa5573e44f59f224e22ca73252c4a9ca65c94199d78eb1e319e989c6c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/89ff0c05b17e1bc16a00f4d6da02a571ac649ff8684f2ac724b8142c81ca860f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/185a786a8ba5448787cbfc0de363a68f56fc1b41c6cfc8698c9a9901cb4f39ac/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/bd77a87c738b14f6a38bd1f72dac906358107c4e55cee096bc74847a90a25b13/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/0b1956e49d8b59fe8f783ce722216eb6cdbde5b2447d602553b86e8f5e981c9e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/77a578030e8e36e0822e33afeba669b2cbea50461e4102a796fd0caf954c6e34/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/999d9278e08e63c7154d9bdd017fcd0b5e8e738cfe219a857f2858985657e1df/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b842ebf4c73b93d4f6db9d30e4461c4937c256de2c3f6e32c8cfc927ed440775/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2582e20b8aa643967902b293ecd0910c632cbb3fdccb733e650e16907131c4ea/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4aa854357c4a6ca3291af243f45d1f43ddbc2bb2fa41917e3f61400f054910e2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7841aa2e5cbc3d886dcb2923529383f673d46b5be88b1f98d04f5545736158a3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/153bdb1d742baf1966579d3190b40189d98c68b260113f53d651e3b788014b4d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/91c3247ed19202469143a744bbe73dbc4dc3aee1ac091c63f968bdc8a6cec9f7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/792155dfc4ff1c85230c68658720f41f7487a8950d7732e1bfeaed5f6bd5c55c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/14f64affd0cc1077d5a86c255e86801dafd49ff16f56f4e75dae19f5531eb36c/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/6b3490c09b675d564737bd71284e9b0555494be375695e32ba8f8613e3fb0cec/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d85ffaa2dc1080de5aeee00a5b6d8a59252372985c2b5ee6e45e987c43adecfe/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0003f4dfbe5ea7dcedfe9b2cc07396913d4a40da4f1165d4eb0e3f17e0f4ce35/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9b28abd5983ae1dd46bd785e2688a00d18909900f54827c073a5abfde42f6bc7/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/9ba896ced7033094674e787da7146dd1bc9e6610df247c383a5f3ced9aa4f230/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1aa97ab3b581b60939e8519aaafe74a53b0c80e63f7fbdba15e507058cf18a3d/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/9d06ef74cd3bda5e9493d5d2cb3d541a543fed618c3906c6f7fee0fbdcc29633/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ce29ef12a78c3dfa522a638474d68131ec8317df8310f5538549cb45355df130/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/6220277fb3e5dbca269af0d6b3ec89c9234d893057f07b80ab4f8a74bc1fb3bd/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/d30c902387890f7d47e567c577b42cc19f7f3e8dc6f4ef025047b75eadce898f/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/51ebd82d18f403408d2a7d223eacf3bb10d6cda157b9c1a368e1283516b57b57/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/8fac1bb7f2d83ffc3086ef2dbf502d40dd300d279b1de000af7063867744bbb5/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/d45b5ea667df444fd5b4693630227225fd5201db7f8c1c4717a206592e4f29e9/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d411c3e4b57df1ab91794b2aabb28770bd8243cefa9aee3a495bd8390139c140/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/19e9eaafab124119669533d73581d5164fdbe1678ab5968ac4a675a63b3ff9da/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/f28d572beb141ee09001b9b9b75bd020b7c80379c960ea2a9cfe9eef8892eb4b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/566a802ebc93333a938c8ef6aa7f579f292d9bf3a34a36a2566fd507c220630e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/623abc006ecfc2883daba3c4a95ba0e2759a55b7088729fc1a677ddc4d0af7c8/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/ea73442da6cddec7f56b80211b764b2bca0b912ab986db50fb30d7061f067a5d/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/768b6562353b7c3096971c5b0da5be38d307d6b52595d8e32f730890ea8b7e42/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/c30c9d256a8957b11b67e1c67c181142f03f3a929c08d2212bde4d4bb0c30199/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/96d59d4f1e2a143a379f3d2c336a3b94d94a7a0f2c42373a497ff130a735845a/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/254076c1bf299cead7ea4d578f38a6237e9b15eedf05f80c6da5460794301fd3/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2316bad749035b000513ff3d2bb99da44814d466afe19d0ac162544f518a2b5a/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/942785f8d6a942d8eadd341014e5cda4ffd1bdbc188309d7fc3486059f8b0380/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6a5461e4ad4ce6100ea0d4524250d37d4c428382315a087ab68f67faa4008aff/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/33ef0fe20f89b074650e01fdd6c11602ef026b2a9a8ad73fd8aefdc2378f1104/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/aec500a41fc10d9ff8e88ea820bc0640434c6c5ea8a19a89a4fc984e1dda6b3b/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/fb0ef7a540b89f83e2348c2f0d95136d78664dd0086b5f34184d183d99b482ef/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/08148b314a8072449f6904394d7a2be140a1ff3ceb5065a36890775e33bef952/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/094e9a1315f99fed091e72d3b05e0c3b7e3b4418038e29c27716f408bee7eb49/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/67c6ddec2ba782c71d185ac63f8c5c4c71fab827e2d9981815ae59c25c88b6f5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/2908a1a6975f61d1f51ff439fb09486904b1dcdf1675d44c941d165ce1278778/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 04 03:17:02.681455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.680851 4959 manager.go:217] Machine: {Timestamp:2026-11-04 03:17:02.680372223 +0000 UTC m=+0.063800260 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118368 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20934aeeddb2377520963ad2beab78 SystemUUID:ec20934a-eedd-b237-7520-963ad2beab78 BootID:faa3cc94-e0e1-4d37-9065-e71a40d736ef Filesystems:[{Device:/var/lib/kubelet/pods/0ed988d6-f52e-4030-8805-1c214876ee88/volumes/kubernetes.io~projected/kube-api-access-pf7dg DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21c7f600e2b63ee8fe3ac2a95ed9908fbad84956781afd00bac84c9ad69afab3/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~projected/kube-api-access-gx8hb DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dab481c8-cff5-4c29-aa52-552434b043bf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1eb395f786650eaace3e61735acbb03ec4164ca0a802294bd36f1d0b44052cd/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0cabe74-608b-461e-a3aa-fa049a28de5b/volumes/kubernetes.io~projected/kube-api-access-vnts4 DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/056b218a-8def-4ac5-8a9b-78b69b08943d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86941193-b41d-47a0-b4cf-ef4221272e45/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/969e817f-6f19-4c15-835c-abcb599d5554/volumes/kubernetes.io~projected/kube-api-access-vl7nn DeviceMajor:0 DeviceMinor:289 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ad24bd15-2732-4e05-9494-ca0691fd9e86/volumes/kubernetes.io~projected/kube-api-access-487vl DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f85ad17a71c5a4508660e1eb2a102d0ebc78f927166c6c0b16bc994ce37b24b9/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b792b02616b4a8e9e4065cc1f33a0d72dc54a9ab385e9ce62ac282b96b17b88d/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/939eba33-05f1-4a74-97e4-3c41074ca9ac/volumes/kubernetes.io~projected/kube-api-access-t5z2j DeviceMajor:0 DeviceMinor:236 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/056b218a-8def-4ac5-8a9b-78b69b08943d/volumes/kubernetes.io~projected/kube-api-access-m4clk DeviceMajor:0 DeviceMinor:239 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d6a461d-3472-4285-b5a9-1a11367f6497/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/284dd9ee-7ea1-4c8d-9674-57c15be65836/volumes/kubernetes.io~projected/kube-api-access-c7mwd DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1/volumes/kubernetes.io~projected/kube-api-access-sdxd5 DeviceMajor:0 DeviceMinor:329 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9891bc-c81c-4ce5-afc9-75c07b82176a/volumes/kubernetes.io~projected/kube-api-access-s2x88 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48/volumes/kubernetes.io~projected/kube-api-access-tkhwc DeviceMajor:0 DeviceMinor:235 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97b3071705d94473c2a601ac47f911d97a3660393989ee59f87f2cf966d2dad5/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4132467-8852-45ae-8c94-bf0d78847bc6/volumes/kubernetes.io~projected/kube-api-access-lns92 DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25be341a9b214e4965a7ba5442f6115f56f5b9def17850c614bcf3ccdf704d8d/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~projected/kube-api-access-gcz2w DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e4b5da75-9530-4e0b-849d-fa50fdec3241/volumes/kubernetes.io~projected/kube-api-access-kcsqq DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7e5fb612ad29e21041d7f14a6b4474e2f423fa87e2760868723e666946ca452/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad271ed185cc1e2c3e66e1cc125cbc021ef131a17eec7d55af0ed80b9b860a1f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa2c5a035e6f65ca6a7b81e6a25ee4f9a2d5d0ecbad4e4ce275c35726f3ec158/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24761c51-4ac8-4976-b04f-91f529dd055e/volumes/kubernetes.io~projected/kube-api-access-9tzvl DeviceMajor:0 DeviceMinor:322 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b24d1578-2e1c-4602-a90e-1d86e1b10645/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/074a2389-db4e-4779-979e-0b9ba07a65bf/volumes/kubernetes.io~projected/kube-api-access-lzppv DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0d2c34775ea931e4830a5f5b7667032aad241bd0cf5fa9ae068dd72b869874a/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3194fa09-26b5-46ff-bada-c1077f024432/volumes/kubernetes.io~projected/kube-api-access-2979t DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9380c359-acb2-43f6-af23-89baa5bee3b4/volumes/kubernetes.io~projected/kube-api-access-fb2r9 DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a736148e0a338ded249ce3b2dd2533d6c12f87b0f84250a37e566f42197f07fb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86941193-b41d-47a0-b4cf-ef4221272e45/volumes/kubernetes.io~projected/kube-api-access-rc2hs DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94d498c93fdaba6a346e7e2eeda97dfc6acee5b5265f36f2dafa182a4d82db80/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c140d95316b4967461aa10a79a2e19041d9403334de8fc559103d94a68384fd5/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6f7f6313aa02a38a5ee7c17ebc47d1e5644e4e070e8b9b576e21574eca18c82/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e04fdd96-e8ac-4338-933c-1b9dadb56b89/volumes/kubernetes.io~projected/kube-api-access-kq9pz DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef2da1733e9f033bc5946d1dac05bfeb7112911dc5fcd1e43754b874d0731b2c/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09ef64076e7b0c503b74dc68b9763e41e0e1f0d44629234d252b91ed84237e31/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a699a906-fae8-47d9-92ba-8eefce6cf724/volumes/kubernetes.io~projected/kube-api-access-74fmz DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/63620800-28b7-4d31-88d8-b75f34a5dda1/volumes/kubernetes.io~projected/kube-api-access-htnz4 DeviceMajor:0 DeviceMinor:238 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/617a4896bb5a543bff4eb8a49ee5db024c7be3e8df5c290171e89a0d532e7f20/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/781b84f6-581f-4357-b447-8d4f92aece81/volumes/kubernetes.io~projected/kube-api-access-4l5sg DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes/kubernetes.io~projected/kube-api-access-9t6hr DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/63620800-28b7-4d31-88d8-b75f34a5dda1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/f762673f-00d6-4182-b7e3-bd16847fd218/volumes/kubernetes.io~projected/kube-api-access-wgkz8 DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45c4d1b188d1cdd7b4d687f79bb09a7ae5ce17ffa7a2876f210908661b081cf8/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c66837ee-2658-4ef6-bee5-d15bbe58ebde/volumes/kubernetes.io~projected/kube-api-access-sd27k DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ed988d6-f52e-4030-8805-1c214876ee88/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6802ea74-e881-496d-b63b-601d10ae7d02/volumes/kubernetes.io~projected/kube-api-access-srm9p DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68892cba95d574b1321399d832b55986a6b2c7b1735a676da4ae55c4a5987ab9/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9fcd5b477cd4c8d5f4aad424046bb72f7c5a67f887a92f96b7d71eceb40c86c/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76ed8641a4a3d4fe475744137f9cd3778adfd96a5e90bcdcfb57034e7c53d83f/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0cabe74-608b-461e-a3aa-fa049a28de5b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/986942d0-73ec-4567-9026-48482bd186ae/volumes/kubernetes.io~projected/kube-api-access-j2z99 DeviceMajor:0 DeviceMinor:237 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:21c7f600e2b63ee MacAddress:12:da:82:4c:d5:52 Speed:10000 Mtu:8901} {Name:45c4d1b188d1cdd MacAddress:06:2d:3a:12:e3:d3 Speed:10000 Mtu:8901} {Name:617a4896bb5a543 MacAddress:fe:26:b5:99:20:c4 Speed:10000 Mtu:8901} {Name:68892cba95d574b MacAddress:82:32:0e:a5:2e:cf Speed:10000 Mtu:8901} {Name:76ed8641a4a3d4f MacAddress:1e:a9:9b:52:b6:bf Speed:10000 Mtu:8901} {Name:94d498c93fdaba6 MacAddress:6e:35:05:64:0e:44 Speed:10000 Mtu:8901} {Name:97b3071705d9447 MacAddress:8e:94:7e:e1:be:32 Speed:10000 Mtu:8901} {Name:a0d2c34775ea931 MacAddress:e2:4a:86:ab:b2:e1 Speed:10000 Mtu:8901} {Name:b792b02616b4a8e MacAddress:4e:c7:8b:9b:43:53 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:30:36:4c:b0:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:0e:e4:a5:d6:ff:52 Speed:0 Mtu:8901} {Name:c140d95316b4967 MacAddress:3a:5e:9a:2a:77:a6 Speed:10000 Mtu:8901} {Name:d6f7f6313aa02a3 MacAddress:c2:61:38:4e:6b:5d Speed:10000 Mtu:8901} {Name:e9fcd5b477cd4c8 MacAddress:2e:60:79:ea:13:f3 Speed:10000 Mtu:8901} {Name:ef2da1733e9f033 MacAddress:46:4e:6d:43:1c:74 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:30:36:4c:b0:ef Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:0e:76:3f:30:52 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:b2:fd:12:5e:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 04 03:17:02.681455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681249 4959 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 04 03:17:02.681455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681298 4959 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 04 03:17:02.681869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681661 4959 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 04 03:17:02.681919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681692 4959 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-184.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 04 03:17:02.681919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681894 4959 topology_manager.go:143] "Creating topology manager with none policy" Nov 04 03:17:02.681919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681907 4959 container_manager_linux.go:311] "Creating device plugin manager" Nov 04 03:17:02.682068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681922 4959 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:17:02.682068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681944 4959 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 04 03:17:02.682068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.681971 4959 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 04 03:17:02.682214 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682194 4959 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 04 03:17:02.682346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682335 4959 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 04 03:17:02.682416 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682383 4959 kubelet.go:493] "Attempting to sync node with API server" Nov 04 03:17:02.682416 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682398 4959 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 04 03:17:02.682416 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682412 4959 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 04 03:17:02.682559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682425 4959 kubelet.go:405] "Adding apiserver pod source" Nov 04 03:17:02.682559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.682437 4959 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 04 03:17:02.683154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.683140 4959 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 03:17:02.683154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.683155 4959 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:17:02.683801 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.683778 4959 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 04 03:17:02.749976 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.749956 4959 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 04 03:17:02.749976 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.749976 4959 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 04 03:17:02.750094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750087 4959 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 04 03:17:02.750294 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750283 4959 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 04 03:17:02.750338 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750313 4959 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 04 03:17:02.750425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750416 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 04 03:17:02.750454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750431 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 04 03:17:02.750454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750437 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 04 03:17:02.750454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750443 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 04 03:17:02.750454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750449 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 04 03:17:02.750454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750454 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750461 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750490 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750498 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750504 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750514 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750523 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750548 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 04 03:17:02.750606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.750556 4959 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 04 03:17:02.754965 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.754937 4959 server.go:1263] "Started kubelet" Nov 04 03:17:02.755091 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.755027 4959 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 04 03:17:02.755452 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.755386 4959 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 04 03:17:02.755585 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.755490 4959 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 04 03:17:02.755526 ip-10-0-132-184 systemd[1]: Started Kubernetes Kubelet. Nov 04 03:17:02.755816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.755796 4959 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 04 03:17:02.756541 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.756523 4959 server.go:317] "Adding debug handlers to kubelet server" Nov 04 03:17:02.759087 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759067 4959 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 04 03:17:02.759187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759084 4959 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 04 03:17:02.759187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759135 4959 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-03 03:11:22 +0000 UTC" deadline="2028-08-06 14:18:39.253036916 +0000 UTC" Nov 04 03:17:02.759187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759173 4959 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15395h1m36.493866479s" Nov 04 03:17:02.759187 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.759136 4959 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:17:02.759187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759186 4959 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 04 03:17:02.759422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759195 4959 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 04 03:17:02.759422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759213 4959 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 04 03:17:02.760426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759976 4959 factory.go:55] Registering systemd factory Nov 04 03:17:02.760426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.759995 4959 factory.go:223] Registration of the systemd container factory successfully Nov 04 03:17:02.760426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.760341 4959 factory.go:153] Registering CRI-O factory Nov 04 03:17:02.760426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.760370 4959 factory.go:223] Registration of the crio container factory successfully Nov 04 03:17:02.762347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.761954 4959 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 04 03:17:02.762347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.761991 4959 factory.go:103] Registering Raw factory Nov 04 03:17:02.762347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.762012 4959 manager.go:1199] Started watching for new ooms in manager Nov 04 03:17:02.762558 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.762513 4959 manager.go:319] Starting recovery of all containers Nov 04 03:17:02.766610 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766571 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63620800-28b7-4d31-88d8-b75f34a5dda1" volumeName="kubernetes.io/projected/63620800-28b7-4d31-88d8-b75f34a5dda1-kube-api-access-htnz4" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766615 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63620800-28b7-4d31-88d8-b75f34a5dda1" volumeName="kubernetes.io/secret/63620800-28b7-4d31-88d8-b75f34a5dda1-serving-cert" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766628 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a699a906-fae8-47d9-92ba-8eefce6cf724" volumeName="kubernetes.io/projected/a699a906-fae8-47d9-92ba-8eefce6cf724-kube-api-access-74fmz" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766640 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0cabe74-608b-461e-a3aa-fa049a28de5b" volumeName="kubernetes.io/projected/d0cabe74-608b-461e-a3aa-fa049a28de5b-kube-api-access-vnts4" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766653 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b5da75-9530-4e0b-849d-fa50fdec3241" volumeName="kubernetes.io/projected/e4b5da75-9530-4e0b-849d-fa50fdec3241-kube-api-access-kcsqq" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766664 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f762673f-00d6-4182-b7e3-bd16847fd218" volumeName="kubernetes.io/projected/f762673f-00d6-4182-b7e3-bd16847fd218-kube-api-access-wgkz8" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766676 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86941193-b41d-47a0-b4cf-ef4221272e45" volumeName="kubernetes.io/projected/86941193-b41d-47a0-b4cf-ef4221272e45-kube-api-access-rc2hs" seLinuxMountContext="" Nov 04 03:17:02.766699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766688 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1" volumeName="kubernetes.io/configmap/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-nginx-conf" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766706 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766719 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66837ee-2658-4ef6-bee5-d15bbe58ebde" volumeName="kubernetes.io/secret/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovn-node-metrics-cert" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766732 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24761c51-4ac8-4976-b04f-91f529dd055e" volumeName="kubernetes.io/projected/24761c51-4ac8-4976-b04f-91f529dd055e-kube-api-access-9tzvl" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766746 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="969e817f-6f19-4c15-835c-abcb599d5554" volumeName="kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766759 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766771 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0cabe74-608b-461e-a3aa-fa049a28de5b" volumeName="kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-trusted-ca-bundle" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766786 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0cabe74-608b-461e-a3aa-fa049a28de5b" volumeName="kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-service-ca-bundle" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766798 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b218a-8def-4ac5-8a9b-78b69b08943d" volumeName="kubernetes.io/projected/056b218a-8def-4ac5-8a9b-78b69b08943d-kube-api-access-m4clk" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766813 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766826 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766837 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66837ee-2658-4ef6-bee5-d15bbe58ebde" volumeName="kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-config" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766850 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4132467-8852-45ae-8c94-bf0d78847bc6" volumeName="kubernetes.io/projected/d4132467-8852-45ae-8c94-bf0d78847bc6-kube-api-access-lns92" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766869 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766881 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ed988d6-f52e-4030-8805-1c214876ee88" volumeName="kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-trusted-ca" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766894 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766906 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad24bd15-2732-4e05-9494-ca0691fd9e86" volumeName="kubernetes.io/projected/ad24bd15-2732-4e05-9494-ca0691fd9e86-kube-api-access-487vl" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766917 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766929 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86941193-b41d-47a0-b4cf-ef4221272e45" volumeName="kubernetes.io/configmap/86941193-b41d-47a0-b4cf-ef4221272e45-config" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766942 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1" volumeName="kubernetes.io/projected/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1-kube-api-access-sdxd5" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766953 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0cabe74-608b-461e-a3aa-fa049a28de5b" volumeName="kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-snapshots" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766971 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e04fdd96-e8ac-4338-933c-1b9dadb56b89" volumeName="kubernetes.io/configmap/e04fdd96-e8ac-4338-933c-1b9dadb56b89-iptables-alerter-script" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766983 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056b218a-8def-4ac5-8a9b-78b69b08943d" volumeName="kubernetes.io/secret/056b218a-8def-4ac5-8a9b-78b69b08943d-cert" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.766995 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ed988d6-f52e-4030-8805-1c214876ee88" volumeName="kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-config" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767007 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="284dd9ee-7ea1-4c8d-9674-57c15be65836" volumeName="kubernetes.io/projected/284dd9ee-7ea1-4c8d-9674-57c15be65836-kube-api-access-c7mwd" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767020 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9380c359-acb2-43f6-af23-89baa5bee3b4" volumeName="kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-cni-binary-copy" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767031 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767046 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="284dd9ee-7ea1-4c8d-9674-57c15be65836" volumeName="kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767059 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767071 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86941193-b41d-47a0-b4cf-ef4221272e45" volumeName="kubernetes.io/secret/86941193-b41d-47a0-b4cf-ef4221272e45-serving-cert" seLinuxMountContext="" Nov 04 03:17:02.767071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767083 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66837ee-2658-4ef6-bee5-d15bbe58ebde" volumeName="kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-script-lib" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767095 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66837ee-2658-4ef6-bee5-d15bbe58ebde" volumeName="kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-env-overrides" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767107 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3194fa09-26b5-46ff-bada-c1077f024432" volumeName="kubernetes.io/projected/3194fa09-26b5-46ff-bada-c1077f024432-kube-api-access-2979t" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767121 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9891bc-c81c-4ce5-afc9-75c07b82176a" volumeName="kubernetes.io/projected/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kube-api-access-s2x88" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767134 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="781b84f6-581f-4357-b447-8d4f92aece81" volumeName="kubernetes.io/projected/781b84f6-581f-4357-b447-8d4f92aece81-kube-api-access-4l5sg" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767148 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6802ea74-e881-496d-b63b-601d10ae7d02" volumeName="kubernetes.io/projected/6802ea74-e881-496d-b63b-601d10ae7d02-kube-api-access-srm9p" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767159 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0cabe74-608b-461e-a3aa-fa049a28de5b" volumeName="kubernetes.io/secret/d0cabe74-608b-461e-a3aa-fa049a28de5b-serving-cert" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767171 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab481c8-cff5-4c29-aa52-552434b043bf" volumeName="kubernetes.io/configmap/dab481c8-cff5-4c29-aa52-552434b043bf-konnectivity-ca" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767182 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="284dd9ee-7ea1-4c8d-9674-57c15be65836" volumeName="kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-sysctl-allowlist" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767194 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6802ea74-e881-496d-b63b-601d10ae7d02" volumeName="kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-default-certificate" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767207 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767220 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24d1578-2e1c-4602-a90e-1d86e1b10645" volumeName="kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-tmp" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767232 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4132467-8852-45ae-8c94-bf0d78847bc6" volumeName="kubernetes.io/empty-dir/d4132467-8852-45ae-8c94-bf0d78847bc6-tmp-dir" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767244 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab481c8-cff5-4c29-aa52-552434b043bf" volumeName="kubernetes.io/secret/dab481c8-cff5-4c29-aa52-552434b043bf-agent-certs" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767258 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="986942d0-73ec-4567-9026-48482bd186ae" volumeName="kubernetes.io/configmap/986942d0-73ec-4567-9026-48482bd186ae-telemetry-config" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767271 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48" volumeName="kubernetes.io/projected/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-kube-api-access-tkhwc" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767283 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e04fdd96-e8ac-4338-933c-1b9dadb56b89" volumeName="kubernetes.io/projected/e04fdd96-e8ac-4338-933c-1b9dadb56b89-kube-api-access-kq9pz" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767295 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ed988d6-f52e-4030-8805-1c214876ee88" volumeName="kubernetes.io/projected/0ed988d6-f52e-4030-8805-1c214876ee88-kube-api-access-pf7dg" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767308 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767346 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c66837ee-2658-4ef6-bee5-d15bbe58ebde" volumeName="kubernetes.io/projected/c66837ee-2658-4ef6-bee5-d15bbe58ebde-kube-api-access-sd27k" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767358 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074a2389-db4e-4779-979e-0b9ba07a65bf" volumeName="kubernetes.io/projected/074a2389-db4e-4779-979e-0b9ba07a65bf-kube-api-access-lzppv" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767372 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63620800-28b7-4d31-88d8-b75f34a5dda1" volumeName="kubernetes.io/configmap/63620800-28b7-4d31-88d8-b75f34a5dda1-config" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767384 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86436dd2-fcc1-4f15-8797-8412a1669011" volumeName="kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767397 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9380c359-acb2-43f6-af23-89baa5bee3b4" volumeName="kubernetes.io/projected/9380c359-acb2-43f6-af23-89baa5bee3b4-kube-api-access-fb2r9" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767408 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767419 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48" volumeName="kubernetes.io/configmap/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-config-volume" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767431 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4b5da75-9530-4e0b-849d-fa50fdec3241" volumeName="kubernetes.io/configmap/e4b5da75-9530-4e0b-849d-fa50fdec3241-serviceca" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767443 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24d1578-2e1c-4602-a90e-1d86e1b10645" volumeName="kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-tuned" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767455 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ed988d6-f52e-4030-8805-1c214876ee88" volumeName="kubernetes.io/secret/0ed988d6-f52e-4030-8805-1c214876ee88-serving-cert" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767500 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9380c359-acb2-43f6-af23-89baa5bee3b4" volumeName="kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-daemon-config" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767515 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad24bd15-2732-4e05-9494-ca0691fd9e86" volumeName="kubernetes.io/configmap/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-config" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767526 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0cabe74-608b-461e-a3aa-fa049a28de5b" volumeName="kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-tmp" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767541 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48" volumeName="kubernetes.io/empty-dir/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-tmp-dir" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767555 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24761c51-4ac8-4976-b04f-91f529dd055e" volumeName="kubernetes.io/empty-dir/24761c51-4ac8-4976-b04f-91f529dd055e-data-volume" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767568 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="284dd9ee-7ea1-4c8d-9674-57c15be65836" volumeName="kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-binary-copy" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767580 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d6a461d-3472-4285-b5a9-1a11367f6497" volumeName="kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767592 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6802ea74-e881-496d-b63b-601d10ae7d02" volumeName="kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-stats-auth" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767605 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="939eba33-05f1-4a74-97e4-3c41074ca9ac" volumeName="kubernetes.io/projected/939eba33-05f1-4a74-97e4-3c41074ca9ac-kube-api-access-t5z2j" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767617 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="986942d0-73ec-4567-9026-48482bd186ae" volumeName="kubernetes.io/projected/986942d0-73ec-4567-9026-48482bd186ae-kube-api-access-j2z99" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767629 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf1881f-7518-49cd-ba73-323e0d05c10e" volumeName="kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767640 4959 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24d1578-2e1c-4602-a90e-1d86e1b10645" volumeName="kubernetes.io/projected/b24d1578-2e1c-4602-a90e-1d86e1b10645-kube-api-access-gx8hb" seLinuxMountContext="" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767653 4959 reconstruct.go:97] "Volume reconstruction finished" Nov 04 03:17:02.768436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.767661 4959 reconciler.go:29] "Reconciler: start to sync state" Nov 04 03:17:02.771658 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.771636 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/1.log" Nov 04 03:17:02.771912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.771885 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/1.log" Nov 04 03:17:02.780498 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.780339 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:17:02.781675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.781573 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:17:02.790104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.789795 4959 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 04 03:17:02.791329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.791303 4959 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 04 03:17:02.791329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.791331 4959 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 04 03:17:02.791505 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.791354 4959 kubelet.go:2521] "Starting kubelet main sync loop" Nov 04 03:17:02.791505 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.791392 4959 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 04 03:17:02.804039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.804009 4959 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="2d307c561a738723d6d4eb2201b580ee003e6c277f75c91fc299a6fb8b256e87" exitCode=0 Nov 04 03:17:02.804039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.804041 4959 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="60d179391a7df18d1cbcb299403477105cb4d62174815ce873f86ae0c326e2a7" exitCode=0 Nov 04 03:17:02.804171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.804052 4959 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="74494edcc789da987900f79efa69514134fdf57fcde75fc493d1290e951c83fa" exitCode=0 Nov 04 03:17:02.804171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.804063 4959 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="0a5c4e8480af28ac9dbaa270d660c8465f9dc8f630da117792a4f8d19bf4277e" exitCode=0 Nov 04 03:17:02.804171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.804072 4959 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="31683b328fc0139a324549e47b7db8e387b0438f9341f918d6416e71896a8bdd" exitCode=0 Nov 04 03:17:02.804171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.804081 4959 generic.go:356] "Generic (PLEG): container finished" podID="284dd9ee-7ea1-4c8d-9674-57c15be65836" containerID="0788dfffd7054f87d8f4dc24909e0cdb0f78f44c139acf4894fc6ee104e13fde" exitCode=0 Nov 04 03:17:02.819817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.819792 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:17:02.820389 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.820360 4959 generic.go:356] "Generic (PLEG): container finished" podID="c66837ee-2658-4ef6-bee5-d15bbe58ebde" containerID="89dade8b81d5c89823e9db759af485d2576466aa708bd258528af613b0caaab6" exitCode=1 Nov 04 03:17:02.842613 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.842579 4959 generic.go:356] "Generic (PLEG): container finished" podID="ee48ef9881478a9b0ef921b39f5cf13c" containerID="e31c21a72b201f4a52d25b3826381397a898d490ca70dde55531f58eaae4f390" exitCode=0 Nov 04 03:17:02.845506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.845484 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/1.log" Nov 04 03:17:02.845593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.845517 4959 generic.go:356] "Generic (PLEG): container finished" podID="0ed988d6-f52e-4030-8805-1c214876ee88" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" exitCode=255 Nov 04 03:17:02.853963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.853946 4959 manager.go:324] Recovery completed Nov 04 03:17:02.859267 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.859247 4959 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:17:02.872629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.872605 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.875112 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.875087 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.875233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.875123 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.875233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.875134 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.877589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877570 4959 cpu_manager.go:226] "Starting" policy="none" Nov 04 03:17:02.877589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877589 4959 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 04 03:17:02.877589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877582 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.877729 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877609 4959 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 04 03:17:02.877783 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877767 4959 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 04 03:17:02.877818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877785 4959 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 04 03:17:02.877818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877803 4959 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 04 03:17:02.877818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877809 4959 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 04 03:17:02.877818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877814 4959 policy_none.go:50] "Start" Nov 04 03:17:02.877920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877824 4959 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 04 03:17:02.877920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877834 4959 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 04 03:17:02.877970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877958 4959 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 04 03:17:02.877970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877965 4959 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 04 03:17:02.877970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.877970 4959 policy_none.go:44] "Start" Nov 04 03:17:02.878972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.878956 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.879028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.878990 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.879028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.879005 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.882409 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.882378 4959 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 04 03:17:02.887515 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.887502 4959 manager.go:341] "Starting Device Plugin manager" Nov 04 03:17:02.887580 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.887529 4959 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 04 03:17:02.887580 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.887538 4959 server.go:87] "Starting device plugin registration server" Nov 04 03:17:02.887861 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.887848 4959 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 04 03:17:02.887923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.887864 4959 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 04 03:17:02.888002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.887982 4959 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 04 03:17:02.888139 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.888099 4959 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 04 03:17:02.888219 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.888151 4959 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 04 03:17:02.888219 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.888158 4959 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 04 03:17:02.888996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.888187 4959 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-04T03:17:02.888137762Z","UUID":"d0024a6b-ea80-45ee-a59a-07f867fe2715","Handler":null,"Name":"","Endpoint":""} Nov 04 03:17:02.889407 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.889380 4959 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 04 03:17:02.889508 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.889414 4959 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" Nov 04 03:17:02.890187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.890169 4959 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 04 03:17:02.890275 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.890194 4959 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 04 03:17:02.891540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.891500 4959 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal"] Nov 04 03:17:02.891682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.891667 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.892593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.892578 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.892686 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.892615 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.892686 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.892631 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.893035 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893017 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.893164 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893150 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.893765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893745 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.893859 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893775 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.893859 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893790 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.893859 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893848 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.893968 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893867 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.893968 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.893877 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.894048 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.894032 4959 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-184.us-west-1.compute.internal\" not found" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.894217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.894070 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"7cf4b9ea98890a60ae37fd39b8c09d27","Type":"ContainerStarted","Data":"e523ccb9e140e301858aa3f2a634f3c1c1d52945b6992144640b68aac002e092"} Nov 04 03:17:02.894397 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.894375 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"7cf4b9ea98890a60ae37fd39b8c09d27","Type":"ContainerStarted","Data":"2c7bf33269d24e2990709e7cd74aecc4ae29963604840a03ec47f257f5e7ff26"} Nov 04 03:17:02.894463 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.894422 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.894539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.894527 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"ee48ef9881478a9b0ef921b39f5cf13c","Type":"ContainerStarted","Data":"8f7b2106a52f2c4b32d6e9939a3b841e5739682c6a0623c5e1f903c28e5aae94"} Nov 04 03:17:02.894586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.894560 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"ee48ef9881478a9b0ef921b39f5cf13c","Type":"ContainerDied","Data":"e31c21a72b201f4a52d25b3826381397a898d490ca70dde55531f58eaae4f390"} Nov 04 03:17:02.894649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.894626 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" event={"ID":"ee48ef9881478a9b0ef921b39f5cf13c","Type":"ContainerStarted","Data":"57a417fadc4b7fb40c754dfc861d0e218782a14814a9b5cd8c61936946e1b3b5"} Nov 04 03:17:02.896785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.896768 4959 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 04 03:17:02.896924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.896899 4959 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:02.896972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.896941 4959 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:02.897862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.897829 4959 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:02.898029 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.898015 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.898105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.898041 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.898105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.898050 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.898208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.898195 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.948663 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.948630 4959 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 04 03:17:02.957043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.957017 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:02.957180 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:02.957073 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.960183 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.960165 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.988113 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.988088 4959 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 04 03:17:02.989155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.989127 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 04 03:17:02.989257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.989166 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 04 03:17:02.989257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.989178 4959 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-184.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 04 03:17:02.989257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.989247 4959 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.997484 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.997418 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.997554 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.997449 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:02.997554 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:02.997537 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cf4b9ea98890a60ae37fd39b8c09d27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"7cf4b9ea98890a60ae37fd39b8c09d27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.032174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.032143 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:03.032303 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:03.032195 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.032303 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.032209 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.095295 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.095259 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:03.095440 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:03.095322 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.098457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.098436 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.098533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.098496 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cf4b9ea98890a60ae37fd39b8c09d27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"7cf4b9ea98890a60ae37fd39b8c09d27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.098564 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.098536 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.098593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.098562 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.098593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.098578 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee48ef9881478a9b0ef921b39f5cf13c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"ee48ef9881478a9b0ef921b39f5cf13c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.098689 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.098628 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cf4b9ea98890a60ae37fd39b8c09d27-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" (UID: \"7cf4b9ea98890a60ae37fd39b8c09d27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.117863 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.117835 4959 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.118002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.117920 4959 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.683357 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.683328 4959 apiserver.go:52] "Watching apiserver" Nov 04 03:17:03.839255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.839230 4959 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 04 03:17:03.839942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.839913 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-v5cxt","openshift-image-registry/image-registry-7685ff7ccc-rbhsz","openshift-insights/insights-operator-7c4986d449-9sfxr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal","openshift-image-registry/node-ca-s9jbx","openshift-insights/insights-runtime-extractor-c44w7","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk","openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74","openshift-network-operator/iptables-alerter-khw7x","openshift-ingress-canary/ingress-canary-zh4vg","openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz","openshift-multus/multus-additional-cni-plugins-j2p7f","openshift-dns/node-resolver-7ztjl","kube-system/global-pull-secret-syncer-cjhm7","kube-system/kas-connection-checker-685895c7c9-7n6xm","openshift-ovn-kubernetes/ovnkube-node-54nv5","openshift-multus/network-metrics-daemon-m9tdv","openshift-network-console/networking-console-plugin-86c478bb88-kx5dm","kube-system/kas-connection-checker-685895c7c9-n2zsd","openshift-multus/multus-svnhr","kube-system/kas-connection-checker-685895c7c9-g5nm2","openshift-dns/dns-default-8cvwl","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg","kube-system/konnectivity-agent-5xngn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp","openshift-network-diagnostics/network-check-target-gxvcp","openshift-ingress/router-default-66bfccb89c-84k2x","openshift-image-registry/image-registry-7f89dfb67b-5spht","openshift-cluster-node-tuning-operator/tuned-8zxlg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc","openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj","openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg"] Nov 04 03:17:03.841062 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.841042 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.841710 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.841691 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:03.842143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.842125 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:03.843939 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.843919 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.844675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.844657 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:03.849659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.849637 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:03.850576 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.850557 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.851048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.851035 4959 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.854170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.854155 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.854631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.854619 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.855655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.855638 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.861035 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.861020 4959 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 04 03:17:03.899055 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899031 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 04 03:17:03.899055 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899031 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 04 03:17:03.899208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899079 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 04 03:17:03.899208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899113 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 04 03:17:03.899208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899119 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 04 03:17:03.899208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899168 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 04 03:17:03.899208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899172 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 04 03:17:03.899429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899245 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 04 03:17:03.899429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899252 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 04 03:17:03.899429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899283 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 04 03:17:03.899429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899324 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2c56k\"" Nov 04 03:17:03.899429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899400 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 04 03:17:03.899429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899423 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4sx77\"" Nov 04 03:17:03.899700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899442 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xgt9d\"" Nov 04 03:17:03.899700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899452 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 04 03:17:03.899700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899532 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r8flv\"" Nov 04 03:17:03.899700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899562 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-57t69\"" Nov 04 03:17:03.899700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.899623 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x7rx6\"" Nov 04 03:17:03.903724 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903700 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.903772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903732 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.903772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903752 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e04fdd96-e8ac-4338-933c-1b9dadb56b89-iptables-alerter-script\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:03.903830 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903789 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2979t\" (UniqueName: \"kubernetes.io/projected/3194fa09-26b5-46ff-bada-c1077f024432-kube-api-access-2979t\") pod \"volume-data-source-validator-58c9f5f46d-pnvkp\" (UID: \"3194fa09-26b5-46ff-bada-c1077f024432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:17:03.903830 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903820 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-netns\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.903929 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903843 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.903929 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903866 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.904016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903941 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-systemd\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.904016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903976 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.904016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.903981 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mwd\" (UniqueName: \"kubernetes.io/projected/284dd9ee-7ea1-4c8d-9674-57c15be65836-kube-api-access-c7mwd\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.904016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904010 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.904203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904040 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fmz\" (UniqueName: \"kubernetes.io/projected/a699a906-fae8-47d9-92ba-8eefce6cf724-kube-api-access-74fmz\") pod \"kas-connection-checker-685895c7c9-n2zsd\" (UID: \"a699a906-fae8-47d9-92ba-8eefce6cf724\") " pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:17:03.904203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904061 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b5da75-9530-4e0b-849d-fa50fdec3241-host\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:03.904203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904079 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:03.904203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-stats-auth\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.904203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904165 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-lib-modules\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.904377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904257 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2z99\" (UniqueName: \"kubernetes.io/projected/986942d0-73ec-4567-9026-48482bd186ae-kube-api-access-j2z99\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:03.904377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904295 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-systemd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.904377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904324 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-stats-auth\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.904377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904326 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.904377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904351 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904384 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzppv\" (UniqueName: \"kubernetes.io/projected/074a2389-db4e-4779-979e-0b9ba07a65bf-kube-api-access-lzppv\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904412 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904433 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904456 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-script-lib\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904516 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904529 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904580 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-kubelet\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.904601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904605 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4132467-8852-45ae-8c94-bf0d78847bc6-tmp-dir\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904622 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm9p\" (UniqueName: \"kubernetes.io/projected/6802ea74-e881-496d-b63b-601d10ae7d02-kube-api-access-srm9p\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904658 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcsqq\" (UniqueName: \"kubernetes.io/projected/e4b5da75-9530-4e0b-849d-fa50fdec3241-kube-api-access-kcsqq\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904684 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzvl\" (UniqueName: \"kubernetes.io/projected/24761c51-4ac8-4976-b04f-91f529dd055e-kube-api-access-9tzvl\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904707 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-node-log\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904729 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-config-volume\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904797 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4132467-8852-45ae-8c94-bf0d78847bc6-tmp-dir\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:03.904840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904829 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dab481c8-cff5-4c29-aa52-552434b043bf-agent-certs\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:17:03.905105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904856 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed988d6-f52e-4030-8805-1c214876ee88-serving-cert\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.905105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904878 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9pz\" (UniqueName: \"kubernetes.io/projected/e04fdd96-e8ac-4338-933c-1b9dadb56b89-kube-api-access-kq9pz\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:03.905105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.904910 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24761c51-4ac8-4976-b04f-91f529dd055e-crio-socket\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.905105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905086 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86941193-b41d-47a0-b4cf-ef4221272e45-config\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:17:03.905105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905104 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0ed988d6-f52e-4030-8805-1c214876ee88-serving-cert\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.905278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905130 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2hs\" (UniqueName: \"kubernetes.io/projected/86941193-b41d-47a0-b4cf-ef4221272e45-kube-api-access-rc2hs\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:17:03.905278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905157 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:03.905278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905182 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-conf\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.905278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905206 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.905278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905238 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86941193-b41d-47a0-b4cf-ef4221272e45-serving-cert\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:17:03.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905316 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905350 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86941193-b41d-47a0-b4cf-ef4221272e45-serving-cert\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:17:03.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905348 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905381 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-cnibin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905396 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-multus\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905448 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/63620800-28b7-4d31-88d8-b75f34a5dda1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:17:03.905690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905513 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhwc\" (UniqueName: \"kubernetes.io/projected/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-kube-api-access-tkhwc\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.905690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905542 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.905690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905567 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-conf-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.905690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905667 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.905864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905703 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-tuned\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.905864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905747 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-socket-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.905864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905780 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:03.905864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905806 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-tuned\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.905864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905809 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905863 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905891 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905916 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-socket-dir-parent\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905939 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-daemon-config\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905961 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-sys-fs\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.905988 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056b218a-8def-4ac5-8a9b-78b69b08943d-cert\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906068 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906070 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-etc-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906121 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906148 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906177 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dab481c8-cff5-4c29-aa52-552434b043bf-konnectivity-ca\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906203 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-k8s-cni-cncf-io\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906220 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906227 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4132467-8852-45ae-8c94-bf0d78847bc6-hosts-file\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:03.906266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906255 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l5sg\" (UniqueName: \"kubernetes.io/projected/781b84f6-581f-4357-b447-8d4f92aece81-kube-api-access-4l5sg\") pod \"kas-connection-checker-685895c7c9-g5nm2\" (UID: \"781b84f6-581f-4357-b447-8d4f92aece81\") " pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906282 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-kubelet\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906303 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-var-lib-kubelet\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906329 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8hb\" (UniqueName: \"kubernetes.io/projected/b24d1578-2e1c-4602-a90e-1d86e1b10645-kube-api-access-gx8hb\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906370 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-device-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906490 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e04fdd96-e8ac-4338-933c-1b9dadb56b89-host-slash\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906556 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-487vl\" (UniqueName: \"kubernetes.io/projected/ad24bd15-2732-4e05-9494-ca0691fd9e86-kube-api-access-487vl\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906586 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysconfig\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906612 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906660 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906696 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-system-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906723 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxd5\" (UniqueName: \"kubernetes.io/projected/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1-kube-api-access-sdxd5\") pod \"migrator-66df5d496b-8sv74\" (UID: \"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:17:03.906738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906739 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-sys\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906757 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906784 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-registration-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906845 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnts4\" (UniqueName: \"kubernetes.io/projected/d0cabe74-608b-461e-a3aa-fa049a28de5b-kube-api-access-vnts4\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906876 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.906900 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907012 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.907047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907031 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-bin\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907227 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907065 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovn-node-metrics-cert\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907227 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907093 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/986942d0-73ec-4567-9026-48482bd186ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:03.907227 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907194 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.907227 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907215 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-config\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907327 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907232 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-kubelet-config\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:03.907327 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907247 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-netns\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.907327 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907280 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-host\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.907327 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907301 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-tmp\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907330 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-config\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907358 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63620800-28b7-4d31-88d8-b75f34a5dda1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907364 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24d1578-2e1c-4602-a90e-1d86e1b10645-tmp\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907384 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0cabe74-608b-461e-a3aa-fa049a28de5b-serving-cert\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907404 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-tmp-dir\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907431 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907455 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-cnibin\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907499 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-tmp-dir\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907518 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/63620800-28b7-4d31-88d8-b75f34a5dda1-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:17:03.907531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907520 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907550 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0cabe74-608b-461e-a3aa-fa049a28de5b-serving-cert\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907555 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-dbus\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907590 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7dg\" (UniqueName: \"kubernetes.io/projected/0ed988d6-f52e-4030-8805-1c214876ee88-kube-api-access-pf7dg\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907615 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24761c51-4ac8-4976-b04f-91f529dd055e-data-volume\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907664 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-var-lib-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907668 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24761c51-4ac8-4976-b04f-91f529dd055e-data-volume\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907705 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-log-socket\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907756 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-netd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907784 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd27k\" (UniqueName: \"kubernetes.io/projected/c66837ee-2658-4ef6-bee5-d15bbe58ebde-kube-api-access-sd27k\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907815 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.907901 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907878 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-env-overrides\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907910 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.907935 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-hostroot\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908006 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908031 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-etc-kubernetes\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908049 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908067 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-cni-binary-copy\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908083 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-run\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908121 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2x88\" (UniqueName: \"kubernetes.io/projected/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kube-api-access-s2x88\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908192 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4clk\" (UniqueName: \"kubernetes.io/projected/056b218a-8def-4ac5-8a9b-78b69b08943d-kube-api-access-m4clk\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:17:03.908217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908219 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-ovn\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908238 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lns92\" (UniqueName: \"kubernetes.io/projected/d4132467-8852-45ae-8c94-bf0d78847bc6-kube-api-access-lns92\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908254 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-default-certificate\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908276 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-multus-certs\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908323 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2r9\" (UniqueName: \"kubernetes.io/projected/9380c359-acb2-43f6-af23-89baa5bee3b4-kube-api-access-fb2r9\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908442 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-default-certificate\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908445 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-kubernetes\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908495 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908537 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnz4\" (UniqueName: \"kubernetes.io/projected/63620800-28b7-4d31-88d8-b75f34a5dda1-kube-api-access-htnz4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908560 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.908594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908584 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908601 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-bin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908616 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-tmp\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908639 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-systemd-units\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908680 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-snapshots\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908725 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-tmp\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908761 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908783 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d0cabe74-608b-461e-a3aa-fa049a28de5b-snapshots\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908793 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-modprobe-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908824 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4b5da75-9530-4e0b-849d-fa50fdec3241-serviceca\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908844 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-system-cni-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908862 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5z2j\" (UniqueName: \"kubernetes.io/projected/939eba33-05f1-4a74-97e4-3c41074ca9ac-kube-api-access-t5z2j\") pod \"network-check-source-6f78d59f99-27kkj\" (UID: \"939eba33-05f1-4a74-97e4-3c41074ca9ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908880 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.908906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908896 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.909268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908923 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkz8\" (UniqueName: \"kubernetes.io/projected/f762673f-00d6-4182-b7e3-bd16847fd218-kube-api-access-wgkz8\") pod \"kas-connection-checker-685895c7c9-7n6xm\" (UID: \"f762673f-00d6-4182-b7e3-bd16847fd218\") " pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:17:03.909268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908950 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.909268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.908969 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.909268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.909065 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-slash\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.909268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.909100 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:03.932701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.932683 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 04 03:17:03.932701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.932687 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8nk47\"" Nov 04 03:17:03.933173 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.933157 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ccs79\"" Nov 04 03:17:03.933633 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.933588 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mprmh\"" Nov 04 03:17:03.933712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.933698 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 04 03:17:03.933712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.933704 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6wxff\"" Nov 04 03:17:03.935944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.935926 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dab481c8-cff5-4c29-aa52-552434b043bf-agent-certs\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:17:03.939176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.939153 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 04 03:17:03.939176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.939162 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.940021 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.940004 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.940500 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.940481 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.940558 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.940514 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.940634 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.940621 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.940982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.940960 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.941026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.941006 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 04 03:17:03.941923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.941907 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.941969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.941924 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 04 03:17:03.941969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.941930 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.942257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942241 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.942257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942255 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.942257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942258 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.942422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942310 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.942422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942382 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.942422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942405 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.942813 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942800 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 04 03:17:03.942862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942836 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.942862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942837 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.942862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942837 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 04 03:17:03.942954 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.942862 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.943054 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943041 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 04 03:17:03.943088 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943077 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 04 03:17:03.943400 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943384 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943392 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943488 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943461 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 04 03:17:03.943531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943501 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 04 03:17:03.943531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943515 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943613 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943666 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 04 03:17:03.943731 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943684 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943731 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943706 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943729 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943741 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943752 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943750 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943784 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943791 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943753 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 04 03:17:03.943825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943803 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.944081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943875 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.944081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943935 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 04 03:17:03.944081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.943984 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s72nt\"" Nov 04 03:17:03.944253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.944238 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e04fdd96-e8ac-4338-933c-1b9dadb56b89-iptables-alerter-script\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:03.945213 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.945199 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.945274 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.945249 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:03.945341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.945331 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:03.945392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.945372 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86941193-b41d-47a0-b4cf-ef4221272e45-config\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:17:03.946483 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.946451 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dab481c8-cff5-4c29-aa52-552434b043bf-konnectivity-ca\") pod \"konnectivity-agent-5xngn\" (UID: \"dab481c8-cff5-4c29-aa52-552434b043bf\") " pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:17:03.946539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.946484 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/63620800-28b7-4d31-88d8-b75f34a5dda1-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:17:03.947588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.947575 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-config\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.947651 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.947584 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovn-node-metrics-cert\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.947651 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.947629 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-config\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.948240 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.948222 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-env-overrides\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.948240 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.948225 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.949550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.949536 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-service-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.949829 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4b5da75-9530-4e0b-849d-fa50fdec3241-serviceca\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.949981 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.950012 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.950318 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5pzz6\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.950521 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5q5qr\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.950775 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7jn7l\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.951144 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c548\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.951201 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qhl58\"" Nov 04 03:17:03.953626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.951219 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9qdq\"" Nov 04 03:17:03.983948 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.983922 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-f24jt\"" Nov 04 03:17:03.984165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.984154 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 04 03:17:03.985559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.985545 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c66837ee-2658-4ef6-bee5-d15bbe58ebde-ovnkube-script-lib\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.991540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.991523 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kfskp\"" Nov 04 03:17:03.992728 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.992713 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztc7x\"" Nov 04 03:17:03.992890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.992876 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 04 03:17:03.993577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.993564 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 04 03:17:03.994194 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994171 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.994194 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994184 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.994343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994286 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 04 03:17:03.994343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994310 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 04 03:17:03.994343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994324 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 04 03:17:03.994343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994337 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.994343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994291 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994349 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994350 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994403 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994413 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994423 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994436 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994450 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994456 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 04 03:17:03.994583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994479 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 04 03:17:03.994975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994611 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmpr4\"" Nov 04 03:17:03.994975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994657 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n89nd\"" Nov 04 03:17:03.994975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994667 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj8mj\"" Nov 04 03:17:03.994975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994685 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2qrjk\"" Nov 04 03:17:03.994975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994744 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:17:03.994975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994966 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sg2zz\"" Nov 04 03:17:03.995233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.994971 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-config-volume\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.995233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.995120 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:03.995233 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:03.995185 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.995359 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.995305 4959 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 04 03:17:03.995805 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:03.995787 4959 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.us-west-1.compute.internal" Nov 04 03:17:03.995974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.995956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9pz\" (UniqueName: \"kubernetes.io/projected/e04fdd96-e8ac-4338-933c-1b9dadb56b89-kube-api-access-kq9pz\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:03.996212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996192 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fmz\" (UniqueName: \"kubernetes.io/projected/a699a906-fae8-47d9-92ba-8eefce6cf724-kube-api-access-74fmz\") pod \"kas-connection-checker-685895c7c9-n2zsd\" (UID: \"a699a906-fae8-47d9-92ba-8eefce6cf724\") " pod="kube-system/kas-connection-checker-685895c7c9-n2zsd" Nov 04 03:17:03.996277 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996232 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056b218a-8def-4ac5-8a9b-78b69b08943d-cert\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:17:03.996374 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996355 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-daemon-config\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.996551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996529 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm9p\" (UniqueName: \"kubernetes.io/projected/6802ea74-e881-496d-b63b-601d10ae7d02-kube-api-access-srm9p\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:03.996764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996741 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mwd\" (UniqueName: \"kubernetes.io/projected/284dd9ee-7ea1-4c8d-9674-57c15be65836-kube-api-access-c7mwd\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.996845 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996827 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.997015 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.996993 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:03.997071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997029 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2979t\" (UniqueName: \"kubernetes.io/projected/3194fa09-26b5-46ff-bada-c1077f024432-kube-api-access-2979t\") pod \"volume-data-source-validator-58c9f5f46d-pnvkp\" (UID: \"3194fa09-26b5-46ff-bada-c1077f024432\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-pnvkp" Nov 04 03:17:03.997294 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997273 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzppv\" (UniqueName: \"kubernetes.io/projected/074a2389-db4e-4779-979e-0b9ba07a65bf-kube-api-access-lzppv\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:03.997351 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997308 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd27k\" (UniqueName: \"kubernetes.io/projected/c66837ee-2658-4ef6-bee5-d15bbe58ebde-kube-api-access-sd27k\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:03.997557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997536 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2hs\" (UniqueName: \"kubernetes.io/projected/86941193-b41d-47a0-b4cf-ef4221272e45-kube-api-access-rc2hs\") pod \"service-ca-operator-544d749c8c-52snz\" (UID: \"86941193-b41d-47a0-b4cf-ef4221272e45\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" Nov 04 03:17:03.997772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997752 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:03.997853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997836 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2r9\" (UniqueName: \"kubernetes.io/projected/9380c359-acb2-43f6-af23-89baa5bee3b4-kube-api-access-fb2r9\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:03.997918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997860 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnz4\" (UniqueName: \"kubernetes.io/projected/63620800-28b7-4d31-88d8-b75f34a5dda1-kube-api-access-htnz4\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg\" (UID: \"63620800-28b7-4d31-88d8-b75f34a5dda1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" Nov 04 03:17:03.997968 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.997935 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.998051 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998031 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzvl\" (UniqueName: \"kubernetes.io/projected/24761c51-4ac8-4976-b04f-91f529dd055e-kube-api-access-9tzvl\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:03.998425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998402 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxd5\" (UniqueName: \"kubernetes.io/projected/c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1-kube-api-access-sdxd5\") pod \"migrator-66df5d496b-8sv74\" (UID: \"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" Nov 04 03:17:03.998653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998625 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4clk\" (UniqueName: \"kubernetes.io/projected/056b218a-8def-4ac5-8a9b-78b69b08943d-kube-api-access-m4clk\") pod \"ingress-canary-zh4vg\" (UID: \"056b218a-8def-4ac5-8a9b-78b69b08943d\") " pod="openshift-ingress-canary/ingress-canary-zh4vg" Nov 04 03:17:03.998725 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998665 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2x88\" (UniqueName: \"kubernetes.io/projected/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kube-api-access-s2x88\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:03.998780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998760 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7nn\" (UniqueName: \"kubernetes.io/projected/969e817f-6f19-4c15-835c-abcb599d5554-kube-api-access-vl7nn\") pod \"network-check-target-gxvcp\" (UID: \"969e817f-6f19-4c15-835c-abcb599d5554\") " pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:17:03.998960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998942 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:03.999017 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.998956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcsqq\" (UniqueName: \"kubernetes.io/projected/e4b5da75-9530-4e0b-849d-fa50fdec3241-kube-api-access-kcsqq\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:03.999297 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999278 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5z2j\" (UniqueName: \"kubernetes.io/projected/939eba33-05f1-4a74-97e4-3c41074ca9ac-kube-api-access-t5z2j\") pod \"network-check-source-6f78d59f99-27kkj\" (UID: \"939eba33-05f1-4a74-97e4-3c41074ca9ac\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-27kkj" Nov 04 03:17:03.999451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999423 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lns92\" (UniqueName: \"kubernetes.io/projected/d4132467-8852-45ae-8c94-bf0d78847bc6-kube-api-access-lns92\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:03.999530 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999486 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l5sg\" (UniqueName: \"kubernetes.io/projected/781b84f6-581f-4357-b447-8d4f92aece81-kube-api-access-4l5sg\") pod \"kas-connection-checker-685895c7c9-g5nm2\" (UID: \"781b84f6-581f-4357-b447-8d4f92aece81\") " pod="kube-system/kas-connection-checker-685895c7c9-g5nm2" Nov 04 03:17:03.999530 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999510 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhwc\" (UniqueName: \"kubernetes.io/projected/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-kube-api-access-tkhwc\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:03.999803 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999778 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 04 03:17:03.999803 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999794 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3d6a461d-3472-4285-b5a9-1a11367f6497-original-pull-secret\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:03.999944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999925 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7dg\" (UniqueName: \"kubernetes.io/projected/0ed988d6-f52e-4030-8805-1c214876ee88-kube-api-access-pf7dg\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:03.999997 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:03.999958 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 04 03:17:04.000101 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000079 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0cabe74-608b-461e-a3aa-fa049a28de5b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:04.000172 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000094 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8hb\" (UniqueName: \"kubernetes.io/projected/b24d1578-2e1c-4602-a90e-1d86e1b10645-kube-api-access-gx8hb\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.000172 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000147 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 04 03:17:04.000276 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000260 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/284dd9ee-7ea1-4c8d-9674-57c15be65836-cni-binary-copy\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.000330 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000269 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:04.000383 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000329 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-487vl\" (UniqueName: \"kubernetes.io/projected/ad24bd15-2732-4e05-9494-ca0691fd9e86-kube-api-access-487vl\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:04.000484 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000442 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9380c359-acb2-43f6-af23-89baa5bee3b4-cni-binary-copy\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.000540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000509 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkz8\" (UniqueName: \"kubernetes.io/projected/f762673f-00d6-4182-b7e3-bd16847fd218-kube-api-access-wgkz8\") pod \"kas-connection-checker-685895c7c9-7n6xm\" (UID: \"f762673f-00d6-4182-b7e3-bd16847fd218\") " pod="kube-system/kas-connection-checker-685895c7c9-7n6xm" Nov 04 03:17:04.000540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000529 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:04.000611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000570 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2z99\" (UniqueName: \"kubernetes.io/projected/986942d0-73ec-4567-9026-48482bd186ae-kube-api-access-j2z99\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:04.000611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000591 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/986942d0-73ec-4567-9026-48482bd186ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:04.000611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000602 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0ed988d6-f52e-4030-8805-1c214876ee88-trusted-ca\") pod \"console-operator-7cd56b8dbc-v5cxt\" (UID: \"0ed988d6-f52e-4030-8805-1c214876ee88\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:04.000774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.000759 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnts4\" (UniqueName: \"kubernetes.io/projected/d0cabe74-608b-461e-a3aa-fa049a28de5b-kube-api-access-vnts4\") pod \"insights-operator-7c4986d449-9sfxr\" (UID: \"d0cabe74-608b-461e-a3aa-fa049a28de5b\") " pod="openshift-insights/insights-operator-7c4986d449-9sfxr" Nov 04 03:17:04.006177 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.006157 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:04.010090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010072 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-run\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010099 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-ovn\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010115 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-multus-certs\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010130 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-kubernetes\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010145 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-bin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010181 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-ovn\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010199 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-kubernetes\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010206 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-run\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010189 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-bin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010205 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-multus-certs\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010217 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-systemd-units\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010245 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-systemd-units\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010252 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010277 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-modprobe-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010302 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-system-cni-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010315 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010330 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-slash\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010353 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-netns\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010372 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010383 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-system-cni-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010390 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-slash\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010392 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010399 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-netns\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010418 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-systemd\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010435 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b5da75-9530-4e0b-849d-fa50fdec3241-host\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010450 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010456 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010416 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-modprobe-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010501 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-lib-modules\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010518 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4b5da75-9530-4e0b-849d-fa50fdec3241-host\") pod \"node-ca-s9jbx\" (UID: \"e4b5da75-9530-4e0b-849d-fa50fdec3241\") " pod="openshift-image-registry/node-ca-s9jbx" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010526 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-systemd\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010457 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010544 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.510524354 +0000 UTC m=+1.893952389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010550 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010563 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-systemd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010584 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-systemd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010567 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010622 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-lib-modules\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010599 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.510587952 +0000 UTC m=+1.894015996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010688 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010713 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.510702505 +0000 UTC m=+1.894130523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010719 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-run-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010744 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-kubelet\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010761 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-node-log\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010779 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24761c51-4ac8-4976-b04f-91f529dd055e-crio-socket\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010795 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-conf\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010796 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-kubelet\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010811 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010828 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-cnibin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.010839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010852 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24761c51-4ac8-4976-b04f-91f529dd055e-crio-socket\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010867 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-node-log\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010867 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-multus\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010898 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-cni-multus\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010900 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010900 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-cnibin\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010931 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-conf\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010935 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-conf-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010959 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-conf-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.010970 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.510962008 +0000 UTC m=+1.894390026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.010991 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011019 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-socket-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011044 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011070 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011095 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011114 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysctl-d\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011116 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-socket-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011118 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011119 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-socket-dir-parent\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011144 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011162 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-sys-fs\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011172 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.511155945 +0000 UTC m=+1.894583976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011180 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011191 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-multus-socket-dir-parent\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011209 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-etc-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011204 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-sys-fs\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011238 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.511222757 +0000 UTC m=+1.894650781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011249 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-etc-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011268 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-k8s-cni-cncf-io\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011294 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4132467-8852-45ae-8c94-bf0d78847bc6-hosts-file\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011307 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-k8s-cni-cncf-io\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011316 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24761c51-4ac8-4976-b04f-91f529dd055e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011319 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-kubelet\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011356 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-var-lib-kubelet\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011361 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4132467-8852-45ae-8c94-bf0d78847bc6-hosts-file\") pod \"node-resolver-7ztjl\" (UID: \"d4132467-8852-45ae-8c94-bf0d78847bc6\") " pod="openshift-dns/node-resolver-7ztjl" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011362 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-var-lib-kubelet\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011376 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-device-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011398 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e04fdd96-e8ac-4338-933c-1b9dadb56b89-host-slash\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011402 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-device-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011414 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysconfig\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011418 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-var-lib-kubelet\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011429 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e04fdd96-e8ac-4338-933c-1b9dadb56b89-host-slash\") pod \"iptables-alerter-khw7x\" (UID: \"e04fdd96-e8ac-4338-933c-1b9dadb56b89\") " pod="openshift-network-operator/iptables-alerter-khw7x" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011435 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011449 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-etc-sysconfig\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011450 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011504 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011515 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011547 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.511535499 +0000 UTC m=+1.894963517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011581 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-system-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011599 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-sys\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011614 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-registration-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011629 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011637 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-system-cni-dir\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011645 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011642 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-sys\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011608 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011675 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011676 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-bin\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011696 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.511685479 +0000 UTC m=+1.895113496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011705 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-registration-dir\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011724 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3e9891bc-c81c-4ce5-afc9-75c07b82176a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mfctc\" (UID: \"3e9891bc-c81c-4ce5-afc9-75c07b82176a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mfctc" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011710 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-bin\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011726 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-kubelet-config\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011751 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-kubelet-config\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011766 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-netns\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011781 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-host\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011808 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011819 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-host-run-netns\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011824 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-cnibin\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011837 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24d1578-2e1c-4602-a90e-1d86e1b10645-host\") pod \"tuned-8zxlg\" (UID: \"b24d1578-2e1c-4602-a90e-1d86e1b10645\") " pod="openshift-cluster-node-tuning-operator/tuned-8zxlg" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011851 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011868 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/284dd9ee-7ea1-4c8d-9674-57c15be65836-cnibin\") pod \"multus-additional-cni-plugins-j2p7f\" (UID: \"284dd9ee-7ea1-4c8d-9674-57c15be65836\") " pod="openshift-multus/multus-additional-cni-plugins-j2p7f" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011873 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-dbus\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011890 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-var-lib-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011893 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011903 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-log-socket\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011923 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-netd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.011932 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.511919201 +0000 UTC m=+1.895347232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : secret "metrics-daemon-secret" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011894 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-run-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011927 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3d6a461d-3472-4285-b5a9-1a11367f6497-dbus\") pod \"global-pull-secret-syncer-cjhm7\" (UID: \"3d6a461d-3472-4285-b5a9-1a11367f6497\") " pod="kube-system/global-pull-secret-syncer-cjhm7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011951 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-log-socket\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011960 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011965 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-cni-netd\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011968 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-var-lib-openvswitch\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.011989 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-hostroot\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.012009 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.012014 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-etc-kubernetes\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.012111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.012038 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.014387 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.012044 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-etc-kubernetes\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.014387 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.012052 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:04.512042729 +0000 UTC m=+1.895470761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:17:04.014387 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.012070 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66837ee-2658-4ef6-bee5-d15bbe58ebde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-54nv5\" (UID: \"c66837ee-2658-4ef6-bee5-d15bbe58ebde\") " pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:04.014387 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.012042 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9380c359-acb2-43f6-af23-89baa5bee3b4-hostroot\") pod \"multus-svnhr\" (UID: \"9380c359-acb2-43f6-af23-89baa5bee3b4\") " pod="openshift-multus/multus-svnhr" Nov 04 03:17:04.144202 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.144178 4959 scope.go:122] "RemoveContainer" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" Nov 04 03:17:04.157755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.157736 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:17:04.516280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516200 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:04.516280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516236 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:04.516280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516258 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516344 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516346 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516372 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516352697 +0000 UTC m=+2.899780716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516407 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516396473 +0000 UTC m=+2.899824511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516423 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516414596 +0000 UTC m=+2.899842615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516403 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516462 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516491 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516517 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516509915 +0000 UTC m=+2.899937933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516507 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516569 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:04.516587 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516596 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516611 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516645 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516679 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516660658 +0000 UTC m=+2.900088689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516689 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516704 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516707 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516725 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516714561 +0000 UTC m=+2.900142602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : secret "metrics-daemon-secret" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516741 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516730967 +0000 UTC m=+2.900158987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516703 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516761 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516745872 +0000 UTC m=+2.900173891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516767 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516779 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.516789 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516812 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516801173 +0000 UTC m=+2.900229203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516830 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:17:04.517105 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.516853 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:05.516847315 +0000 UTC m=+2.900275365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:17:04.662562 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.662514 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:04.854975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.854950 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:17:04.855402 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.855310 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/1.log" Nov 04 03:17:04.855402 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.855343 4959 generic.go:356] "Generic (PLEG): container finished" podID="0ed988d6-f52e-4030-8805-1c214876ee88" containerID="ae7706af2e97dcb60492670ce00ea7d6f40b4deb75f914f85217838739512e27" exitCode=255 Nov 04 03:17:04.855568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.855424 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" event={"ID":"0ed988d6-f52e-4030-8805-1c214876ee88","Type":"ContainerDied","Data":"ae7706af2e97dcb60492670ce00ea7d6f40b4deb75f914f85217838739512e27"} Nov 04 03:17:04.855568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.855517 4959 scope.go:122] "RemoveContainer" containerID="6f6c2a75e200234fcfe8b0ff5740b4375456dc5011d46f80468b06591a22fe80" Nov 04 03:17:04.855982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:04.855959 4959 scope.go:122] "RemoveContainer" containerID="ae7706af2e97dcb60492670ce00ea7d6f40b4deb75f914f85217838739512e27" Nov 04 03:17:04.856139 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:04.856120 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5cxt_openshift-console-operator(0ed988d6-f52e-4030-8805-1c214876ee88)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" Nov 04 03:17:05.526674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526581 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:05.526674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526621 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:05.526674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526651 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526701 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526731 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526735 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526763 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526733 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526801 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526804 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526810 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.526797119 +0000 UTC m=+4.910225158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526878 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.526864403 +0000 UTC m=+4.910292421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526905 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526943 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526954 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.526987 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.526979333 +0000 UTC m=+4.910407382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.526961 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527001 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.526992831 +0000 UTC m=+4.910420851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527013 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:17:05.526974 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527023 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.52701761 +0000 UTC m=+4.910445628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527036 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.527030657 +0000 UTC m=+4.910458675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527046 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527050 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.527043106 +0000 UTC m=+4.910471130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527059 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.527107 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527113 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.527103877 +0000 UTC m=+4.910531914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.527137 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527152 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527177 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.527167009 +0000 UTC m=+4.910595065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : secret "metrics-daemon-secret" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527221 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:17:05.527661 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.527259 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:07.527252323 +0000 UTC m=+4.910680355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:17:05.814649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.814568 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:05.860231 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.860192 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8sv74" event={"ID":"c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1","Type":"ContainerStarted","Data":"9ca2229b0a740f2f42561182c4a736c611f64cdb3475d47cb5ad3a708a97a306"} Nov 04 03:17:05.861774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.861756 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:17:05.862042 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:05.862028 4959 scope.go:122] "RemoveContainer" containerID="ae7706af2e97dcb60492670ce00ea7d6f40b4deb75f914f85217838739512e27" Nov 04 03:17:05.862202 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:05.862185 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5cxt_openshift-console-operator(0ed988d6-f52e-4030-8805-1c214876ee88)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" Nov 04 03:17:06.744824 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:06.744793 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7ztjl_d4132467-8852-45ae-8c94-bf0d78847bc6/dns-node-resolver/0.log" Nov 04 03:17:06.938952 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:06.938920 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:07.545654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.545619 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:07.545654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.545653 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.545673 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545769 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545780 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545801 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7685ff7ccc-rbhsz: secret "image-registry-tls" not found Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545813 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.545801012 +0000 UTC m=+8.929229043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545837 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:17:07.545895 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545866 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls podName:86436dd2-fcc1-4f15-8797-8412a1669011 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.54584878 +0000 UTC m=+8.929276801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls") pod "image-registry-7685ff7ccc-rbhsz" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011") : secret "image-registry-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.545895 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.545938 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545965 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.545955231 +0000 UTC m=+8.929383250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.545986 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.545977667 +0000 UTC m=+8.929405692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.546009 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546013 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546056 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546085 4959 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546088 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.546073169 +0000 UTC m=+8.929501224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546094 4959 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f89dfb67b-5spht: secret "image-registry-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.546046 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546117 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.546104945 +0000 UTC m=+8.929532968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:17:07.546155 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546155 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls podName:acf1881f-7518-49cd-ba73-323e0d05c10e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.546139477 +0000 UTC m=+8.929567513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls") pod "image-registry-7f89dfb67b-5spht" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e") : secret "image-registry-tls" not found Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.546171 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.546200 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.546226 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546263 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546278 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546288 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.546281203 +0000 UTC m=+8.929709222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546300 4959 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546304 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.546294846 +0000 UTC m=+8.929722892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : secret "metrics-daemon-secret" not found Nov 04 03:17:07.546559 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:07.546338 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls podName:24761c51-4ac8-4976-b04f-91f529dd055e nodeName:}" failed. No retries permitted until 2026-11-04 03:17:11.546330883 +0000 UTC m=+8.929758935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c44w7" (UID: "24761c51-4ac8-4976-b04f-91f529dd055e") : secret "insights-runtime-extractor-tls" not found Nov 04 03:17:07.944603 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:07.944577 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9jbx_e4b5da75-9530-4e0b-849d-fa50fdec3241/node-ca/0.log" Nov 04 03:17:08.744549 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:08.744520 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zh4vg_056b218a-8def-4ac5-8a9b-78b69b08943d/serve-healthcheck-canary/0.log" Nov 04 03:17:08.753629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:08.753596 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:08.945456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:08.945402 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8sv74_c3fe2170-93d8-4926-b4c3-3d0bdfb6abe1/migrator/0.log" Nov 04 03:17:09.044411 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:09.044309 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:09.296199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:09.296107 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:09.346431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:09.346403 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg_63620800-28b7-4d31-88d8-b75f34a5dda1/kube-storage-version-migrator-operator/0.log" Nov 04 03:17:09.559078 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:09.558999 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:09.913747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:09.913713 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:10.160745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:10.160640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:10.564870 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:10.564838 4959 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:10.565225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:10.565211 4959 scope.go:122] "RemoveContainer" containerID="ae7706af2e97dcb60492670ce00ea7d6f40b4deb75f914f85217838739512e27" Nov 04 03:17:10.565398 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:10.565380 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-v5cxt_openshift-console-operator(0ed988d6-f52e-4030-8805-1c214876ee88)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" Nov 04 03:17:10.716986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:10.716954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:11.134280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.134232 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:11.380199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.380164 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:11.414752 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.414686 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:11.414889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.414874 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:11.582787 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.582751 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.582790 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.582822 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.582858 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.582883 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.582921 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.582900685 +0000 UTC m=+16.966328719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : configmap references non-existent config key: service-ca.crt Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.582921 4959 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:11.582971 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.582965 4959 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.582989 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls podName:986942d0-73ec-4567-9026-48482bd186ae nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.582970635 +0000 UTC m=+16.966398658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xlwwg" (UID: "986942d0-73ec-4567-9026-48482bd186ae") : secret "cluster-monitoring-operator-tls" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.582989 4959 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.583016 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583049 4959 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583046 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls podName:ad24bd15-2732-4e05-9494-ca0691fd9e86 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.58303245 +0000 UTC m=+16.966460482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-ss6mk" (UID: "ad24bd15-2732-4e05-9494-ca0691fd9e86") : secret "samples-operator-tls" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583119 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs podName:6802ea74-e881-496d-b63b-601d10ae7d02 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.583107324 +0000 UTC m=+16.966535347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs") pod "router-default-66bfccb89c-84k2x" (UID: "6802ea74-e881-496d-b63b-601d10ae7d02") : secret "router-metrics-certs-default" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583137 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs podName:074a2389-db4e-4779-979e-0b9ba07a65bf nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.583128941 +0000 UTC m=+16.966556959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs") pod "network-metrics-daemon-m9tdv" (UID: "074a2389-db4e-4779-979e-0b9ba07a65bf") : secret "metrics-daemon-secret" not found Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.583191 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.583277 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.583306 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.583302 4959 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 04 03:17:11.583336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.583338 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:11.583932 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583378 4959 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 04 03:17:11.583932 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583425 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls podName:bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.583409618 +0000 UTC m=+16.966837656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls") pod "dns-default-8cvwl" (UID: "bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48") : secret "dns-default-metrics-tls" not found Nov 04 03:17:11.583932 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583511 4959 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 04 03:17:11.583932 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:11.583560 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert podName:8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1 nodeName:}" failed. No retries permitted until 2026-11-04 03:17:19.583549839 +0000 UTC m=+16.966977870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kx5dm" (UID: "8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1") : secret "networking-console-plugin-cert" not found Nov 04 03:17:11.585774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.585744 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24761c51-4ac8-4976-b04f-91f529dd055e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c44w7\" (UID: \"24761c51-4ac8-4976-b04f-91f529dd055e\") " pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:11.585774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.585763 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"image-registry-7685ff7ccc-rbhsz\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:11.585914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.585819 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"image-registry-7f89dfb67b-5spht\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:11.626222 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.626196 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:11.644223 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.644198 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:11.655410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.655185 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:11.656249 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.656224 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c44w7" Nov 04 03:17:11.922868 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:11.922796 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:12.073963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.073929 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c44w7"] Nov 04 03:17:12.077744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.077721 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7685ff7ccc-rbhsz"] Nov 04 03:17:12.078301 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:12.078278 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24761c51_4ac8_4976_b04f_91f529dd055e.slice/crio-0c649f8703eb7ff6a8d2f98f700df4f53eb8c6107d7a03a5b28352971b7673e2 WatchSource:0}: Error finding container 0c649f8703eb7ff6a8d2f98f700df4f53eb8c6107d7a03a5b28352971b7673e2: Status 404 returned error can't find the container with id 0c649f8703eb7ff6a8d2f98f700df4f53eb8c6107d7a03a5b28352971b7673e2 Nov 04 03:17:12.106608 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:12.106582 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86436dd2_fcc1_4f15_8797_8412a1669011.slice/crio-4a4fa6ecf7c039e96b6af111539ad1f0a31157b758a7b6dc524cfe6897a00cc4 WatchSource:0}: Error finding container 4a4fa6ecf7c039e96b6af111539ad1f0a31157b758a7b6dc524cfe6897a00cc4: Status 404 returned error can't find the container with id 4a4fa6ecf7c039e96b6af111539ad1f0a31157b758a7b6dc524cfe6897a00cc4 Nov 04 03:17:12.126051 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.126027 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f89dfb67b-5spht"] Nov 04 03:17:12.129827 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:12.129803 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf1881f_7518_49cd_ba73_323e0d05c10e.slice/crio-e26593c186a0bd40bab2ac677172ece57b1b76b7a05c74145227bfda24829464 WatchSource:0}: Error finding container e26593c186a0bd40bab2ac677172ece57b1b76b7a05c74145227bfda24829464: Status 404 returned error can't find the container with id e26593c186a0bd40bab2ac677172ece57b1b76b7a05c74145227bfda24829464 Nov 04 03:17:12.723735 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.723689 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:12.886326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.886287 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" event={"ID":"86436dd2-fcc1-4f15-8797-8412a1669011","Type":"ContainerStarted","Data":"92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353"} Nov 04 03:17:12.886539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.886339 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" event={"ID":"86436dd2-fcc1-4f15-8797-8412a1669011","Type":"ContainerStarted","Data":"4a4fa6ecf7c039e96b6af111539ad1f0a31157b758a7b6dc524cfe6897a00cc4"} Nov 04 03:17:12.887819 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.887792 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c44w7" event={"ID":"24761c51-4ac8-4976-b04f-91f529dd055e","Type":"ContainerStarted","Data":"af1d822c294d2f98c9368306550ba90b4feeef7b7d7689f4a83e5a81dc1c16e0"} Nov 04 03:17:12.887905 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.887843 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c44w7" event={"ID":"24761c51-4ac8-4976-b04f-91f529dd055e","Type":"ContainerStarted","Data":"0c649f8703eb7ff6a8d2f98f700df4f53eb8c6107d7a03a5b28352971b7673e2"} Nov 04 03:17:12.889198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.889177 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" event={"ID":"acf1881f-7518-49cd-ba73-323e0d05c10e","Type":"ContainerStarted","Data":"4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31"} Nov 04 03:17:12.889260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.889212 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" event={"ID":"acf1881f-7518-49cd-ba73-323e0d05c10e","Type":"ContainerStarted","Data":"e26593c186a0bd40bab2ac677172ece57b1b76b7a05c74145227bfda24829464"} Nov 04 03:17:12.989311 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:12.989237 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:17:13.118932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.118897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:13.411184 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.411147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:13.659362 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.659331 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:13.894772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.894749 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:13.894772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.894756 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c44w7" event={"ID":"24761c51-4ac8-4976-b04f-91f529dd055e","Type":"ContainerStarted","Data":"09bb544fe7fdab59dd43a77a8b732dde10e21364117d89ef2f598850baf9c2de"} Nov 04 03:17:13.895185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.894800 4959 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 04 03:17:13.919732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:13.919686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:14.436413 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:14.436378 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:14.634505 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:14.634452 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:14.884355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:14.884314 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:15.133190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.133153 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:15.376919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.376889 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:15.630061 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.629976 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:15.792593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.792558 4959 scope.go:122] "RemoveContainer" containerID="ae7706af2e97dcb60492670ce00ea7d6f40b4deb75f914f85217838739512e27" Nov 04 03:17:15.904030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.903944 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c44w7" event={"ID":"24761c51-4ac8-4976-b04f-91f529dd055e","Type":"ContainerStarted","Data":"bc2f1e4373b9c114c57d75359a13afefb30268149e51dd595b777ab0707504ee"} Nov 04 03:17:15.905657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.905641 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:17:15.905740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.905711 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" event={"ID":"0ed988d6-f52e-4030-8805-1c214876ee88","Type":"ContainerStarted","Data":"5c04816aa6354a2330a253f2b6e738ba85cd1b7521a7955c2d60d40c0869eea7"} Nov 04 03:17:15.905963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.905945 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:15.906921 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.906899 4959 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-v5cxt container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Nov 04 03:17:15.907017 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.906942 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" podUID="0ed988d6-f52e-4030-8805-1c214876ee88" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Nov 04 03:17:15.919368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:15.919344 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:16.353089 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:16.353058 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:17:16.718396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:16.718308 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:16.913632 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:16.913604 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-v5cxt" Nov 04 03:17:17.126650 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:17.126613 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:17.224261 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:17.224235 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:17.420322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:17.420241 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:17.668624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:17.668588 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:17.918300 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:17.918272 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:18.356395 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:18.356360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:18.717086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:18.716991 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:18.959282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:18.959245 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:19.318655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.318620 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:19.569105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.569035 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:19.657006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.656976 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:19.657127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657013 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:19.657127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657057 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:19.657127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657076 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:19.657127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657092 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:19.657127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657121 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:19.657421 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657148 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:19.657864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.657838 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6802ea74-e881-496d-b63b-601d10ae7d02-service-ca-bundle\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:19.659963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.659905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24bd15-2732-4e05-9494-ca0691fd9e86-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-ss6mk\" (UID: \"ad24bd15-2732-4e05-9494-ca0691fd9e86\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:19.659963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.659944 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6802ea74-e881-496d-b63b-601d10ae7d02-metrics-certs\") pod \"router-default-66bfccb89c-84k2x\" (UID: \"6802ea74-e881-496d-b63b-601d10ae7d02\") " pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:19.660120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.660080 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/986942d0-73ec-4567-9026-48482bd186ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xlwwg\" (UID: \"986942d0-73ec-4567-9026-48482bd186ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:19.660120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.660080 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48-metrics-tls\") pod \"dns-default-8cvwl\" (UID: \"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48\") " pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:19.660366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.660346 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/074a2389-db4e-4779-979e-0b9ba07a65bf-metrics-certs\") pod \"network-metrics-daemon-m9tdv\" (UID: \"074a2389-db4e-4779-979e-0b9ba07a65bf\") " pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:19.660366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.660355 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kx5dm\" (UID: \"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:19.742427 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.742398 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:19.742427 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.742418 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" Nov 04 03:17:19.742810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.742496 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" Nov 04 03:17:19.744877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.744861 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" Nov 04 03:17:19.750369 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.750347 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tdv" Nov 04 03:17:19.755593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.755525 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:19.920988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:19.920935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:20.165757 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.165720 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:20.195353 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.195332 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66bfccb89c-84k2x"] Nov 04 03:17:20.197841 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:20.197814 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6802ea74_e881_496d_b63b_601d10ae7d02.slice/crio-c3242a38edaf09297cd529ddc815248b04ab75f72c8d99172f3e9dcbcfa53813 WatchSource:0}: Error finding container c3242a38edaf09297cd529ddc815248b04ab75f72c8d99172f3e9dcbcfa53813: Status 404 returned error can't find the container with id c3242a38edaf09297cd529ddc815248b04ab75f72c8d99172f3e9dcbcfa53813 Nov 04 03:17:20.238864 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:20.238839 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986942d0_73ec_4567_9026_48482bd186ae.slice/crio-e02f92a1481d6ca604518dce29108e954854862975ec8e4bf4a1e2948816b158 WatchSource:0}: Error finding container e02f92a1481d6ca604518dce29108e954854862975ec8e4bf4a1e2948816b158: Status 404 returned error can't find the container with id e02f92a1481d6ca604518dce29108e954854862975ec8e4bf4a1e2948816b158 Nov 04 03:17:20.246052 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.246031 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg"] Nov 04 03:17:20.247356 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.246908 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk"] Nov 04 03:17:20.247987 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.247933 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m9tdv"] Nov 04 03:17:20.250325 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:20.250300 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdcd7d1_cc9a_4fa5_8082_2cd25247e3c1.slice/crio-8820e365bf647338695e9b2be0cc536f34b6c099b606d594038c462be602ebfd WatchSource:0}: Error finding container 8820e365bf647338695e9b2be0cc536f34b6c099b606d594038c462be602ebfd: Status 404 returned error can't find the container with id 8820e365bf647338695e9b2be0cc536f34b6c099b606d594038c462be602ebfd Nov 04 03:17:20.262487 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:20.262447 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074a2389_db4e_4779_979e_0b9ba07a65bf.slice/crio-de9b1fb9f55bb564efe6f360dd8114850e57725be8840ee9c558499d3fc793b7 WatchSource:0}: Error finding container de9b1fb9f55bb564efe6f360dd8114850e57725be8840ee9c558499d3fc793b7: Status 404 returned error can't find the container with id de9b1fb9f55bb564efe6f360dd8114850e57725be8840ee9c558499d3fc793b7 Nov 04 03:17:20.296243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.296220 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kx5dm"] Nov 04 03:17:20.296777 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.296759 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8cvwl"] Nov 04 03:17:20.922269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.922232 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" event={"ID":"ad24bd15-2732-4e05-9494-ca0691fd9e86","Type":"ContainerStarted","Data":"f9f3099e3af397fc9e537625fb19279a5b68cc2f4c3f2dc93fd533d34f60044b"} Nov 04 03:17:20.923381 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.923356 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8cvwl" event={"ID":"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48","Type":"ContainerStarted","Data":"22777074831fc8a62ef3aec16453e154800cf24ef64161bcbd637ade291f497b"} Nov 04 03:17:20.924793 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.924767 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66bfccb89c-84k2x" event={"ID":"6802ea74-e881-496d-b63b-601d10ae7d02","Type":"ContainerStarted","Data":"3992a8b3959b3b8367ee92b1048c1ca90cbaf49108ca686213b4ed1e624e9f00"} Nov 04 03:17:20.924892 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.924813 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66bfccb89c-84k2x" event={"ID":"6802ea74-e881-496d-b63b-601d10ae7d02","Type":"ContainerStarted","Data":"c3242a38edaf09297cd529ddc815248b04ab75f72c8d99172f3e9dcbcfa53813"} Nov 04 03:17:20.925988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.925970 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" event={"ID":"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1","Type":"ContainerStarted","Data":"8820e365bf647338695e9b2be0cc536f34b6c099b606d594038c462be602ebfd"} Nov 04 03:17:20.926994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.926976 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m9tdv" event={"ID":"074a2389-db4e-4779-979e-0b9ba07a65bf","Type":"ContainerStarted","Data":"de9b1fb9f55bb564efe6f360dd8114850e57725be8840ee9c558499d3fc793b7"} Nov 04 03:17:20.928034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:20.928018 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" event={"ID":"986942d0-73ec-4567-9026-48482bd186ae","Type":"ContainerStarted","Data":"e02f92a1481d6ca604518dce29108e954854862975ec8e4bf4a1e2948816b158"} Nov 04 03:17:21.742938 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:21.742769 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:21.746267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:21.746246 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:21.779104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:21.779047 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-54nv5" Nov 04 03:17:21.934202 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:21.934155 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:21.935403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:21.935330 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66bfccb89c-84k2x" Nov 04 03:17:22.939756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:22.939554 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kx5dm" event={"ID":"8cdcd7d1-cc9a-4fa5-8082-2cd25247e3c1","Type":"ContainerStarted","Data":"3ec0a5ddfd18da2592f4dc16fed590fc760cd4c3d56f09d03570a3fd0332e761"} Nov 04 03:17:22.942856 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:22.942826 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m9tdv" event={"ID":"074a2389-db4e-4779-979e-0b9ba07a65bf","Type":"ContainerStarted","Data":"c42fb9e17c0cefeb55765f04233651eea594cdea1e907a7adf799a6f7a491fc5"} Nov 04 03:17:22.943004 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:22.942877 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m9tdv" event={"ID":"074a2389-db4e-4779-979e-0b9ba07a65bf","Type":"ContainerStarted","Data":"0126aaf0086c9fe71a5510455760345e6f2441044b7ac22250cbc65e744ce24f"} Nov 04 03:17:23.752525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.752491 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:23.948049 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.948017 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" event={"ID":"ad24bd15-2732-4e05-9494-ca0691fd9e86","Type":"ContainerStarted","Data":"32df0f4266730f65543013809b738cbfef9396551e9e23842ff16af05ab74538"} Nov 04 03:17:23.948485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.948094 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-ss6mk" event={"ID":"ad24bd15-2732-4e05-9494-ca0691fd9e86","Type":"ContainerStarted","Data":"1d583ca05e76aef7bd0c2faf9792f6e2145d6221f7c3d95dfd7866b92787431f"} Nov 04 03:17:23.949974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.949948 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8cvwl" event={"ID":"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48","Type":"ContainerStarted","Data":"d6aa27dfd9c3fee56f20bb5dc1906ee4cabcda60748def8aa665153f788ad135"} Nov 04 03:17:23.950080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.949991 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8cvwl" event={"ID":"bb5fd6b8-7b8c-4d71-ae63-98377fbf7b48","Type":"ContainerStarted","Data":"63bb499971de2f5736f4574c53f82337766c05b5977bb73c287b2254948de5c8"} Nov 04 03:17:23.950080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.950037 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:23.951519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:23.951493 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xlwwg" event={"ID":"986942d0-73ec-4567-9026-48482bd186ae","Type":"ContainerStarted","Data":"77a4ab005794e08bff0ba297bf2fee1a9951478fe716ac792025d5919768cb06"} Nov 04 03:17:24.044784 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:24.044708 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:24.450550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:24.450501 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:24.723626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:24.723535 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:24.785363 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:24.785333 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7685ff7ccc-rbhsz"] Nov 04 03:17:25.024911 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:25.024808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:25.277285 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:25.277198 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:25.545103 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:25.545007 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:25.786800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:25.786770 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:27.231560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:27.231524 4959 patch_prober.go:29] interesting pod/image-registry-7f89dfb67b-5spht container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 03:17:27.231923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:27.231580 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:27.716298 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:27.716266 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5xngn" Nov 04 03:17:28.836253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:28.836218 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:29.090749 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:29.090668 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:29.340528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:29.340497 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:32.589179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:32.589142 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:32.837115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:32.837073 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:32.991524 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:32.991442 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gxvcp" Nov 04 03:17:33.958120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:33.958093 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8cvwl" Nov 04 03:17:33.983740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:33.983680 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:34.234065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.233984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:34.486292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.486213 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:34.656309 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.656276 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jhbb6"] Nov 04 03:17:34.682032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.682006 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.742545 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.742462 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:34.786745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786717 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786754 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786778 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-wtmp\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786881 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-sys\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786928 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-root\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786949 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x46j\" (UniqueName: \"kubernetes.io/projected/06463d76-3838-4853-aab5-ae5c5ed6e275-kube-api-access-4x46j\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786967 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-tls\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.786995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.786992 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06463d76-3838-4853-aab5-ae5c5ed6e275-metrics-client-ca\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.787185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.787060 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-textfile\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.787855 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.787829 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 04 03:17:34.788031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.788014 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krstf\"" Nov 04 03:17:34.788134 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.788079 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 04 03:17:34.791748 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.791726 4959 patch_prober.go:29] interesting pod/image-registry-7685ff7ccc-rbhsz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 03:17:34.791858 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.791783 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" podUID="86436dd2-fcc1-4f15-8797-8412a1669011" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:34.806154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.806130 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 04 03:17:34.838653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.838629 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 04 03:17:34.887788 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887755 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-sys\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.887900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887794 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-root\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.887900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887821 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x46j\" (UniqueName: \"kubernetes.io/projected/06463d76-3838-4853-aab5-ae5c5ed6e275-kube-api-access-4x46j\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.887900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887849 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-tls\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.887900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887862 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-sys\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.887900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887881 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-root\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.887900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887891 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06463d76-3838-4853-aab5-ae5c5ed6e275-metrics-client-ca\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887938 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-textfile\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.887977 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.888105 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.888154 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-wtmp\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.888310 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-textfile\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.888315 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-wtmp\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888604 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.888578 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.888604 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.888590 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06463d76-3838-4853-aab5-ae5c5ed6e275-metrics-client-ca\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.890465 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.890445 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.890539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.890448 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/06463d76-3838-4853-aab5-ae5c5ed6e275-node-exporter-tls\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:34.902336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.902317 4959 patch_prober.go:29] interesting pod/image-registry-7f89dfb67b-5spht container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 03:17:34.902442 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.902354 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:34.949711 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:34.949685 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x46j\" (UniqueName: \"kubernetes.io/projected/06463d76-3838-4853-aab5-ae5c5ed6e275-kube-api-access-4x46j\") pod \"node-exporter-jhbb6\" (UID: \"06463d76-3838-4853-aab5-ae5c5ed6e275\") " pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:35.029596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:35.029516 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jhbb6" Nov 04 03:17:35.034037 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:35.034011 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:35.064726 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:17:35.064695 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06463d76_3838_4853_aab5_ae5c5ed6e275.slice/crio-014880846c8731d10990918a23468d4cb402f0967967881c2e5ff3ec04f1719e WatchSource:0}: Error finding container 014880846c8731d10990918a23468d4cb402f0967967881c2e5ff3ec04f1719e: Status 404 returned error can't find the container with id 014880846c8731d10990918a23468d4cb402f0967967881c2e5ff3ec04f1719e Nov 04 03:17:35.278991 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:35.278961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:35.528319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:35.528282 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:35.998705 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:35.998669 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbb6" event={"ID":"06463d76-3838-4853-aab5-ae5c5ed6e275","Type":"ContainerStarted","Data":"014880846c8731d10990918a23468d4cb402f0967967881c2e5ff3ec04f1719e"} Nov 04 03:17:36.045445 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:36.045405 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:36.300117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:36.300024 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:36.552164 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:36.552085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:36.970894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:36.970856 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:37.231875 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:37.231810 4959 patch_prober.go:29] interesting pod/image-registry-7f89dfb67b-5spht container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 04 03:17:37.231875 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:37.231860 4959 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:17:38.009432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:38.009400 4959 generic.go:356] "Generic (PLEG): container finished" podID="06463d76-3838-4853-aab5-ae5c5ed6e275" containerID="1e4a91b73d56b4e8a9daf1c07a17f414da4c4f66b0a1503be8bf5021f627664f" exitCode=0 Nov 04 03:17:38.009630 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:38.009490 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbb6" event={"ID":"06463d76-3838-4853-aab5-ae5c5ed6e275","Type":"ContainerDied","Data":"1e4a91b73d56b4e8a9daf1c07a17f414da4c4f66b0a1503be8bf5021f627664f"} Nov 04 03:17:38.797237 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:38.797200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:39.015616 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.015586 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbb6" event={"ID":"06463d76-3838-4853-aab5-ae5c5ed6e275","Type":"ContainerStarted","Data":"0412eda5ecbe086046ce075c65f98ffebd29b28a6ffbd03585cdb04bc28dd356"} Nov 04 03:17:39.015802 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.015640 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbb6" event={"ID":"06463d76-3838-4853-aab5-ae5c5ed6e275","Type":"ContainerStarted","Data":"86fffa1ff9c0ee284373b81e031c0c73f802218d94b4e99393e9b9b05b04e3bb"} Nov 04 03:17:39.095316 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.095285 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:39.189626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.189568 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jhbb6" podStartSLOduration=3.269091258 podStartE2EDuration="5.189553674s" totalImagesPullingTime="1.83757482s" totalInitContainerRuntime="82.887596ms" isStatefulPod=true podCreationTimestamp="2026-11-04 03:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:17:39.189428309 +0000 UTC m=+36.572856351" watchObservedRunningTime="2026-11-04 03:17:39.189553674 +0000 UTC m=+36.572981714" Nov 04 03:17:39.348520 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.348390 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:39.603287 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.603205 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:39.853282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:39.853235 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:41.720821 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:41.720739 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:42.341315 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:42.341268 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:42.587539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:42.587507 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:43.329065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:43.329009 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:43.635813 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:43.635716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:43.926996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:43.926916 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:44.790383 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:44.790357 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:44.902747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:44.902721 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:17:45.731932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:45.731896 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:46.030567 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:46.030486 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:46.354287 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:46.354253 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:17:46.735265 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:46.735187 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:46.993065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:46.992970 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:47.209855 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:47.209828 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f89dfb67b-5spht"] Nov 04 03:17:47.428727 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:47.428698 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:47.679607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:47.679525 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:47.924307 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:47.924272 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:48.985017 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:48.984982 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:49.805770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:49.805716 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" podUID="86436dd2-fcc1-4f15-8797-8412a1669011" containerName="registry" containerID="cri-o://92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353" gracePeriod=30 Nov 04 03:17:50.036934 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.036901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:50.046559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.046540 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:50.060346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.060280 4959 generic.go:356] "Generic (PLEG): container finished" podID="86436dd2-fcc1-4f15-8797-8412a1669011" containerID="92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353" exitCode=0 Nov 04 03:17:50.060346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.060328 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" Nov 04 03:17:50.060522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.060341 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" event={"ID":"86436dd2-fcc1-4f15-8797-8412a1669011","Type":"ContainerDied","Data":"92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353"} Nov 04 03:17:50.060522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.060392 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7685ff7ccc-rbhsz" event={"ID":"86436dd2-fcc1-4f15-8797-8412a1669011","Type":"ContainerDied","Data":"4a4fa6ecf7c039e96b6af111539ad1f0a31157b758a7b6dc524cfe6897a00cc4"} Nov 04 03:17:50.060522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.060421 4959 scope.go:122] "RemoveContainer" containerID="92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353" Nov 04 03:17:50.068367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.068344 4959 scope.go:122] "RemoveContainer" containerID="92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353" Nov 04 03:17:50.068675 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:17:50.068654 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353\": container with ID starting with 92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353 not found: ID does not exist" containerID="92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353" Nov 04 03:17:50.068740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.068685 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353"} err="failed to get container status \"92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353\": rpc error: code = NotFound desc = could not find container \"92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353\": container with ID starting with 92d56828747c92e2fb83c8edd8860b51cd84d6ffc41d251d140f6a0113147353 not found: ID does not exist" Nov 04 03:17:50.131099 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131065 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131113 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131144 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131160 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131189 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131210 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131526 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131263 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131526 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131325 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") pod \"86436dd2-fcc1-4f15-8797-8412a1669011\" (UID: \"86436dd2-fcc1-4f15-8797-8412a1669011\") " Nov 04 03:17:50.131886 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131854 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:50.132000 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.131979 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:17:50.134250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.134222 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:50.134348 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.134280 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:50.134425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.134391 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:17:50.134425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.134413 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "kube-api-access-9t6hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:50.134425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.134418 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:17:50.138561 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.138447 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted" pod "86436dd2-fcc1-4f15-8797-8412a1669011" (UID: "86436dd2-fcc1-4f15-8797-8412a1669011"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:17:50.232379 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232345 4959 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-bound-sa-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232379 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232381 4959 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-image-registry-private-configuration\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232391 4959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-trusted-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232400 4959 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-registry-tls\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232410 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6hr\" (UniqueName: \"kubernetes.io/projected/86436dd2-fcc1-4f15-8797-8412a1669011-kube-api-access-9t6hr\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232419 4959 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86436dd2-fcc1-4f15-8797-8412a1669011-installation-pull-secrets\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232428 4959 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86436dd2-fcc1-4f15-8797-8412a1669011-registry-certificates\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.232614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.232436 4959 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86436dd2-fcc1-4f15-8797-8412a1669011-ca-trust-extracted\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:17:50.480619 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.480587 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7685ff7ccc-rbhsz"] Nov 04 03:17:50.484762 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.484739 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7685ff7ccc-rbhsz"] Nov 04 03:17:50.796391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:50.796319 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86436dd2-fcc1-4f15-8797-8412a1669011" path="/var/lib/kubelet/pods/86436dd2-fcc1-4f15-8797-8412a1669011/volumes" Nov 04 03:17:51.123960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:51.123924 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:51.416848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:51.416758 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:51.669092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:51.669005 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:52.649922 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:52.649895 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:52.932202 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:52.932110 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:53.745760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:53.745729 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:54.041345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:54.041255 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:17:55.127652 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:17:55.127610 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:05.608539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:05.608502 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:05.854983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:05.854951 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:12.232323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.232257 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" containerID="cri-o://4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31" gracePeriod=30 Nov 04 03:18:12.462454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.462422 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:12.485503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.485434 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:18:12.504736 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.504694 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.504923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.504761 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.504923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.504855 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.504923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.504884 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.505085 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.504924 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.505085 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.504959 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.505085 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.505026 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.505085 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.505061 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") pod \"acf1881f-7518-49cd-ba73-323e0d05c10e\" (UID: \"acf1881f-7518-49cd-ba73-323e0d05c10e\") " Nov 04 03:18:12.505298 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.505181 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:12.507299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.507146 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:18:12.509527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.508374 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:12.509527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.508450 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:12.509527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.509264 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:12.511725 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.509933 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:18:12.513908 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.513842 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "kube-api-access-gcz2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:18:12.515234 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.515209 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted" pod "acf1881f-7518-49cd-ba73-323e0d05c10e" (UID: "acf1881f-7518-49cd-ba73-323e0d05c10e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:18:12.605590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605559 4959 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/acf1881f-7518-49cd-ba73-323e0d05c10e-ca-trust-extracted\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605585 4959 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-bound-sa-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605595 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcz2w\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-kube-api-access-gcz2w\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605606 4959 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-certificates\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605616 4959 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/acf1881f-7518-49cd-ba73-323e0d05c10e-registry-tls\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605625 4959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acf1881f-7518-49cd-ba73-323e0d05c10e-trusted-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605633 4959 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-installation-pull-secrets\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.605812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.605642 4959 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/acf1881f-7518-49cd-ba73-323e0d05c10e-image-registry-private-configuration\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:18:12.721073 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:12.721034 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:13.014482 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.014444 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:13.152391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.152356 4959 generic.go:356] "Generic (PLEG): container finished" podID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerID="4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31" exitCode=0 Nov 04 03:18:13.152596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.152412 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" Nov 04 03:18:13.152596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.152442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" event={"ID":"acf1881f-7518-49cd-ba73-323e0d05c10e","Type":"ContainerDied","Data":"4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31"} Nov 04 03:18:13.152596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.152513 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f89dfb67b-5spht" event={"ID":"acf1881f-7518-49cd-ba73-323e0d05c10e","Type":"ContainerDied","Data":"e26593c186a0bd40bab2ac677172ece57b1b76b7a05c74145227bfda24829464"} Nov 04 03:18:13.152596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.152534 4959 scope.go:122] "RemoveContainer" containerID="4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31" Nov 04 03:18:13.160525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.160508 4959 scope.go:122] "RemoveContainer" containerID="4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31" Nov 04 03:18:13.160827 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:18:13.160809 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31\": container with ID starting with 4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31 not found: ID does not exist" containerID="4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31" Nov 04 03:18:13.160872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.160837 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31"} err="failed to get container status \"4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31\": rpc error: code = NotFound desc = could not find container \"4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31\": container with ID starting with 4ccbf22754455d10ad4f5d64176f04da893389aaf11d116ecdd55ed74b3c8d31 not found: ID does not exist" Nov 04 03:18:13.262214 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.262176 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:13.318863 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.318835 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f89dfb67b-5spht"] Nov 04 03:18:13.369072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.369045 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f89dfb67b-5spht"] Nov 04 03:18:13.540995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:13.540915 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:14.157204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:14.157170 4959 generic.go:356] "Generic (PLEG): container finished" podID="63620800-28b7-4d31-88d8-b75f34a5dda1" containerID="b938883b564ddd4b19a452be62e8a8fc9043a41b8a46e5a2650e8b1cb99bb68a" exitCode=0 Nov 04 03:18:14.157438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:14.157244 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" event={"ID":"63620800-28b7-4d31-88d8-b75f34a5dda1","Type":"ContainerDied","Data":"b938883b564ddd4b19a452be62e8a8fc9043a41b8a46e5a2650e8b1cb99bb68a"} Nov 04 03:18:14.157649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:14.157632 4959 scope.go:122] "RemoveContainer" containerID="b938883b564ddd4b19a452be62e8a8fc9043a41b8a46e5a2650e8b1cb99bb68a" Nov 04 03:18:14.797208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:14.797170 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" path="/var/lib/kubelet/pods/acf1881f-7518-49cd-ba73-323e0d05c10e/volumes" Nov 04 03:18:15.163407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:15.163369 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-r4zkg" event={"ID":"63620800-28b7-4d31-88d8-b75f34a5dda1","Type":"ContainerStarted","Data":"9bd6fd908d61ffdbbfee3cd69bb09fb1c643b456f1f33314f3f290c4ad67a1aa"} Nov 04 03:18:16.354185 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:18:16.354150 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:18:23.751245 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:23.751198 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:24.000717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:24.000687 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:25.189815 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:25.189780 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:25.438453 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:25.438414 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:29.216282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:29.216245 4959 generic.go:356] "Generic (PLEG): container finished" podID="86941193-b41d-47a0-b4cf-ef4221272e45" containerID="a19f86492e0203f4e7221dc4abbbeb6e55c4991e48ef9f641cf60e4ed6b1be4f" exitCode=0 Nov 04 03:18:29.216702 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:29.216324 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" event={"ID":"86941193-b41d-47a0-b4cf-ef4221272e45","Type":"ContainerDied","Data":"a19f86492e0203f4e7221dc4abbbeb6e55c4991e48ef9f641cf60e4ed6b1be4f"} Nov 04 03:18:29.216702 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:29.216683 4959 scope.go:122] "RemoveContainer" containerID="a19f86492e0203f4e7221dc4abbbeb6e55c4991e48ef9f641cf60e4ed6b1be4f" Nov 04 03:18:30.221175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:30.221144 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-52snz" event={"ID":"86941193-b41d-47a0-b4cf-ef4221272e45","Type":"ContainerStarted","Data":"bee24cf0cf5ed13bfade05eb9824c75f6a1c3232b331cb9598bf8c890525ab63"} Nov 04 03:18:33.233882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:33.233850 4959 generic.go:356] "Generic (PLEG): container finished" podID="d0cabe74-608b-461e-a3aa-fa049a28de5b" containerID="1a398c2fece9be60931f30f76910cf2d67c67e073550bfb5af3796cb571d57aa" exitCode=0 Nov 04 03:18:33.234253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:33.233899 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" event={"ID":"d0cabe74-608b-461e-a3aa-fa049a28de5b","Type":"ContainerDied","Data":"1a398c2fece9be60931f30f76910cf2d67c67e073550bfb5af3796cb571d57aa"} Nov 04 03:18:33.234253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:33.234188 4959 scope.go:122] "RemoveContainer" containerID="1a398c2fece9be60931f30f76910cf2d67c67e073550bfb5af3796cb571d57aa" Nov 04 03:18:34.239762 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:34.239721 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-9sfxr" event={"ID":"d0cabe74-608b-461e-a3aa-fa049a28de5b","Type":"ContainerStarted","Data":"f60d0093652b9ab152fffe08fd92ce5c6f90b1c606b8ed6adff49513a41bf950"} Nov 04 03:18:38.754815 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:38.754771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:39.052216 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:39.052136 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:46.353796 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:18:46.353761 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:18:53.023713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:53.023678 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:18:54.890040 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:18:54.889997 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:19:09.915698 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:19:09.915656 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:19:11.302617 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:19:11.302525 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:19:12.369697 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:19:12.369661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:19:16.354672 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:19:16.354635 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:19:45.221917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:19:45.221883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:19:46.353276 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:19:46.353240 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:19:46.507044 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:19:46.507006 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:19:53.344299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:19:53.344261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:16.353306 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:20:16.353267 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:20:46.318359 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:46.318270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:46.356439 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:20:46.356403 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:20:46.632531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:46.632464 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:46.940430 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:46.940347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:48.113116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113086 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz"] Nov 04 03:20:48.113582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113368 4959 cpu_manager.go:414] "containerMap: removing container" podUID="86436dd2-fcc1-4f15-8797-8412a1669011" containerName="registry" Nov 04 03:20:48.113582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113379 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86436dd2-fcc1-4f15-8797-8412a1669011" containerName="registry" Nov 04 03:20:48.113582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113402 4959 cpu_manager.go:414] "containerMap: removing container" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" Nov 04 03:20:48.113582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113408 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" Nov 04 03:20:48.113582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113461 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf1881f-7518-49cd-ba73-323e0d05c10e" containerName="registry" Nov 04 03:20:48.113582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.113488 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="86436dd2-fcc1-4f15-8797-8412a1669011" containerName="registry" Nov 04 03:20:48.116334 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.116318 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.172685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.172651 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 04 03:20:48.172855 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.172707 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 04 03:20:48.172855 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.172712 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 04 03:20:48.172855 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.172717 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 04 03:20:48.172964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.172931 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-szrgb\"" Nov 04 03:20:48.221032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.221003 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz"] Nov 04 03:20:48.229457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.229428 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:48.255641 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255616 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.255776 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255679 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.255776 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255745 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.255776 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255765 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.255876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255794 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.255876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255811 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8glz\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.255939 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.255884 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357042 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357249 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357094 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357249 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357207 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357249 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357240 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357285 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357308 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8glz\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357338 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.357749 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.357723 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.359790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.359771 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.359896 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.359875 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.359972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.359956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.360011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.359966 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.426710 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.426646 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.426829 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.426793 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8glz\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz\") pod \"istiod-openshift-gateway-7876bfcb74-f4mfz\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.427433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.427419 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:20:48.569726 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.569695 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:48.848680 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:48.848646 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz"] Nov 04 03:20:48.851651 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:20:48.851617 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e61f3eb_4169_48e2_8c70_7969754480d6.slice/crio-71b95b5450f8bb79151bd08cde30e8e45895fd05176eae47a12856a0cbaa2334 WatchSource:0}: Error finding container 71b95b5450f8bb79151bd08cde30e8e45895fd05176eae47a12856a0cbaa2334: Status 404 returned error can't find the container with id 71b95b5450f8bb79151bd08cde30e8e45895fd05176eae47a12856a0cbaa2334 Nov 04 03:20:49.577642 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:49.577605 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:20:49.708248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:49.708208 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" event={"ID":"9e61f3eb-4169-48e2-8c70-7969754480d6","Type":"ContainerStarted","Data":"71b95b5450f8bb79151bd08cde30e8e45895fd05176eae47a12856a0cbaa2334"} Nov 04 03:20:53.687758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:20:53.687721 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:13.769995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:13.769955 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:21:13.770324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:13.770024 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:21:14.809685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:14.809639 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" event={"ID":"9e61f3eb-4169-48e2-8c70-7969754480d6","Type":"ContainerStarted","Data":"2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195"} Nov 04 03:21:14.810184 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:14.809729 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:21:14.811498 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:14.811459 4959 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-f4mfz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:21:14.811629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:14.811518 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" podUID="9e61f3eb-4169-48e2-8c70-7969754480d6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:21:15.021158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:15.021088 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" podStartSLOduration=2.105447072 podStartE2EDuration="27.021074382s" totalImagesPullingTime="24.91562731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:21:15.020716463 +0000 UTC m=+252.404144504" watchObservedRunningTime="2026-11-04 03:21:15.021074382 +0000 UTC m=+252.404502422" Nov 04 03:21:15.248229 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:15.248191 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:15.492491 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:15.492448 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:15.755808 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:15.755768 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:15.813664 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:15.813636 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:21:16.084173 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:16.084140 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:16.353396 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:21:16.353317 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:21:16.414094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:16.414061 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:19.737452 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:19.737415 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:29.737121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:29.737090 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:30.673650 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.673622 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld"] Nov 04 03:21:30.676840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.676825 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.731046 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.731012 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 04 03:21:30.731167 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.731083 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fqc69\"" Nov 04 03:21:30.780942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.780915 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 04 03:21:30.780942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.780916 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 04 03:21:30.781331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.781118 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99ab2797-d9ea-40d8-8cda-4ecaba436837-manifests\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.781331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.781151 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjq4\" (UniqueName: \"kubernetes.io/projected/99ab2797-d9ea-40d8-8cda-4ecaba436837-kube-api-access-hzjq4\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.781331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.781172 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-metrics-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.781331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.781229 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.781331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.781327 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ab2797-d9ea-40d8-8cda-4ecaba436837-tmp\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.781536 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.781352 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-webhook-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.794941 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.794919 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 04 03:21:30.795045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.794976 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 04 03:21:30.882183 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882141 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-webhook-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.882183 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882185 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99ab2797-d9ea-40d8-8cda-4ecaba436837-manifests\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.882444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882209 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjq4\" (UniqueName: \"kubernetes.io/projected/99ab2797-d9ea-40d8-8cda-4ecaba436837-kube-api-access-hzjq4\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.882444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882226 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-metrics-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.882444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882246 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.882444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882295 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ab2797-d9ea-40d8-8cda-4ecaba436837-tmp\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.882755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.882733 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99ab2797-d9ea-40d8-8cda-4ecaba436837-manifests\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.884926 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.884905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.885025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.885000 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-metrics-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.885120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.885100 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99ab2797-d9ea-40d8-8cda-4ecaba436837-tmp\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.885200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.885186 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99ab2797-d9ea-40d8-8cda-4ecaba436837-webhook-cert\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.944049 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.943990 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjq4\" (UniqueName: \"kubernetes.io/projected/99ab2797-d9ea-40d8-8cda-4ecaba436837-kube-api-access-hzjq4\") pod \"opendatahub-operator-controller-manager-549fcdc555-dzzld\" (UID: \"99ab2797-d9ea-40d8-8cda-4ecaba436837\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:30.991613 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:30.991577 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:31.376366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:31.376339 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld"] Nov 04 03:21:31.774817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:31.774783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:32.093284 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:21:32.093254 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ab2797_d9ea_40d8_8cda_4ecaba436837.slice/crio-80d4831d19abc176a48894f4c6c060065bef956eaacd271cc00a112967a694d6 WatchSource:0}: Error finding container 80d4831d19abc176a48894f4c6c060065bef956eaacd271cc00a112967a694d6: Status 404 returned error can't find the container with id 80d4831d19abc176a48894f4c6c060065bef956eaacd271cc00a112967a694d6 Nov 04 03:21:32.141729 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:32.141700 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld"] Nov 04 03:21:32.876857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:32.876820 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" event={"ID":"99ab2797-d9ea-40d8-8cda-4ecaba436837","Type":"ContainerStarted","Data":"80d4831d19abc176a48894f4c6c060065bef956eaacd271cc00a112967a694d6"} Nov 04 03:21:33.337970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:33.337901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:33.658292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:33.658192 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:33.923524 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:33.923419 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:35.891762 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:35.891722 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" event={"ID":"99ab2797-d9ea-40d8-8cda-4ecaba436837","Type":"ContainerStarted","Data":"31bd0a8288671afbf5d178f3639aef7c69ca5b7def60afc14fe9a6b016d9d5f3"} Nov 04 03:21:36.897457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:36.897414 4959 generic.go:356] "Generic (PLEG): container finished" podID="99ab2797-d9ea-40d8-8cda-4ecaba436837" containerID="31bd0a8288671afbf5d178f3639aef7c69ca5b7def60afc14fe9a6b016d9d5f3" exitCode=0 Nov 04 03:21:36.897865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:36.897501 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" event={"ID":"99ab2797-d9ea-40d8-8cda-4ecaba436837","Type":"ContainerDied","Data":"31bd0a8288671afbf5d178f3639aef7c69ca5b7def60afc14fe9a6b016d9d5f3"} Nov 04 03:21:37.903617 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:37.903583 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" event={"ID":"99ab2797-d9ea-40d8-8cda-4ecaba436837","Type":"ContainerStarted","Data":"6b3d2c559aae7fc1c61f4fff482788258d0c5ebff5891e8e7c951e51f12cdc9f"} Nov 04 03:21:37.904021 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:37.903726 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:38.756984 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:38.756948 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:39.581876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:39.581842 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:39.891913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:39.891821 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:42.076816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:42.076778 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:42.378708 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:42.378625 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:43.923432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:43.923397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:44.185485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:44.185383 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:46.354611 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:21:46.354577 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:21:46.986215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:46.986181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:47.911830 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:47.911788 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:48.910381 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:48.910355 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" Nov 04 03:21:49.159732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:49.159679 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:49.232396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:49.232293 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-549fcdc555-dzzld" podStartSLOduration=14.614926617 podStartE2EDuration="19.232278922s" totalImagesPullingTime="4.206077076s" totalInitContainerRuntime="411.275229ms" isStatefulPod=false podCreationTimestamp="2026-11-04 03:21:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:21:38.166397565 +0000 UTC m=+275.549825618" watchObservedRunningTime="2026-11-04 03:21:49.232278922 +0000 UTC m=+286.615706962" Nov 04 03:21:53.709422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:53.709385 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:54.005907 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:54.005827 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:54.283116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:54.283033 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:56.129333 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:56.129299 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:56.395745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:56.395650 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:58.683061 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:58.683024 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:59.063726 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:59.063625 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:59.391991 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:59.391952 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:21:59.928364 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:21:59.928328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:00.235518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:00.235421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:00.500890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:00.500797 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:01.234800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:01.234764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:01.576900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:01.576861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:02.000785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.000702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:02.292448 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.292354 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:02.572662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.572574 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:02.801234 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.801204 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:22:02.801391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.801336 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:22:02.808106 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.808085 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:22:02.808241 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.808147 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:22:02.826639 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:02.826612 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:03.360600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:03.360559 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:03.620680 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:03.620591 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:05.735084 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:05.735053 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:06.047680 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.047598 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8"] Nov 04 03:22:06.055851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.055828 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.073147 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.073121 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25b3c6fc-bb59-4110-9088-580d91d2aa51-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5rhg8\" (UID: \"25b3c6fc-bb59-4110-9088-580d91d2aa51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.073275 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.073155 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmszz\" (UniqueName: \"kubernetes.io/projected/25b3c6fc-bb59-4110-9088-580d91d2aa51-kube-api-access-gmszz\") pod \"cert-manager-webhook-5dbbbf75cc-5rhg8\" (UID: \"25b3c6fc-bb59-4110-9088-580d91d2aa51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.147280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.147240 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8"] Nov 04 03:22:06.174117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.174086 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25b3c6fc-bb59-4110-9088-580d91d2aa51-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5rhg8\" (UID: \"25b3c6fc-bb59-4110-9088-580d91d2aa51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.174306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.174123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmszz\" (UniqueName: \"kubernetes.io/projected/25b3c6fc-bb59-4110-9088-580d91d2aa51-kube-api-access-gmszz\") pod \"cert-manager-webhook-5dbbbf75cc-5rhg8\" (UID: \"25b3c6fc-bb59-4110-9088-580d91d2aa51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.196114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.196089 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ghk2n\"" Nov 04 03:22:06.196114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.196089 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 04 03:22:06.196226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.196128 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 04 03:22:06.205330 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.205308 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:06.239542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.239511 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25b3c6fc-bb59-4110-9088-580d91d2aa51-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-5rhg8\" (UID: \"25b3c6fc-bb59-4110-9088-580d91d2aa51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.248783 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.248760 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmszz\" (UniqueName: \"kubernetes.io/projected/25b3c6fc-bb59-4110-9088-580d91d2aa51-kube-api-access-gmszz\") pod \"cert-manager-webhook-5dbbbf75cc-5rhg8\" (UID: \"25b3c6fc-bb59-4110-9088-580d91d2aa51\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.380175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.380136 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:06.758851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.758775 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8"] Nov 04 03:22:06.762020 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:22:06.761991 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25b3c6fc_bb59_4110_9088_580d91d2aa51.slice/crio-6f6b6b467888a4a5075fe9252506dfd284bbf6e80adccded9d1ca2973909611a WatchSource:0}: Error finding container 6f6b6b467888a4a5075fe9252506dfd284bbf6e80adccded9d1ca2973909611a: Status 404 returned error can't find the container with id 6f6b6b467888a4a5075fe9252506dfd284bbf6e80adccded9d1ca2973909611a Nov 04 03:22:06.763986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:06.763970 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:22:07.017866 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:07.017785 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" event={"ID":"25b3c6fc-bb59-4110-9088-580d91d2aa51","Type":"ContainerStarted","Data":"6f6b6b467888a4a5075fe9252506dfd284bbf6e80adccded9d1ca2973909611a"} Nov 04 03:22:09.329787 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:09.329749 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:09.726222 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:09.726130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:10.023017 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:10.022939 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:10.280215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:10.280122 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:10.531632 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:10.531547 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:11.035564 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:11.035538 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" event={"ID":"25b3c6fc-bb59-4110-9088-580d91d2aa51","Type":"ContainerStarted","Data":"60675ff0f1c76783fa29b9038b309db336e122f152d4530b2971389313fe647b"} Nov 04 03:22:11.035682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:11.035665 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:11.160971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:11.160908 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" podStartSLOduration=1.947718929 podStartE2EDuration="6.160891041s" totalImagesPullingTime="4.213172112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:11.160324048 +0000 UTC m=+308.543752084" watchObservedRunningTime="2026-11-04 03:22:11.160891041 +0000 UTC m=+308.544319081" Nov 04 03:22:11.612510 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:11.612449 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:16.198996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:16.198954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:16.354439 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:22:16.354398 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:22:16.450367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:16.450292 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:17.042560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:17.042527 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-5rhg8" Nov 04 03:22:18.017584 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.017552 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:22:18.020821 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.020805 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.037158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.037130 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:18.068247 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.068210 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hq47\" (UniqueName: \"kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47\") pod \"limitador-limitador-74dfcfb894-wsld5\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.068404 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.068312 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file\") pod \"limitador-limitador-74dfcfb894-wsld5\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.077858 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.077789 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 04 03:22:18.078225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.078201 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 04 03:22:18.125609 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.125578 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 04 03:22:18.125609 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.125578 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rhb6m\"" Nov 04 03:22:18.126764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.126744 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:22:18.169362 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.169328 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file\") pod \"limitador-limitador-74dfcfb894-wsld5\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.169588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.169391 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hq47\" (UniqueName: \"kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47\") pod \"limitador-limitador-74dfcfb894-wsld5\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.170042 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.170017 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file\") pod \"limitador-limitador-74dfcfb894-wsld5\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.230885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.230850 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hq47\" (UniqueName: \"kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47\") pod \"limitador-limitador-74dfcfb894-wsld5\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.264715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.264680 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:22:18.332608 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.332580 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:18.704499 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:18.704442 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:22:18.707174 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:22:18.707145 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977397c0_4ef8_4724_a10e_95e85fde7208.slice/crio-21f87b7b462bfd6ec2581ee8b499062e5fc51dcfe48a166fbfdf16a8c55feb93 WatchSource:0}: Error finding container 21f87b7b462bfd6ec2581ee8b499062e5fc51dcfe48a166fbfdf16a8c55feb93: Status 404 returned error can't find the container with id 21f87b7b462bfd6ec2581ee8b499062e5fc51dcfe48a166fbfdf16a8c55feb93 Nov 04 03:22:19.068127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.068035 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" event={"ID":"977397c0-4ef8-4724-a10e-95e85fde7208","Type":"ContainerStarted","Data":"21f87b7b462bfd6ec2581ee8b499062e5fc51dcfe48a166fbfdf16a8c55feb93"} Nov 04 03:22:19.320920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.320845 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-kkx5w"] Nov 04 03:22:19.325077 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.325058 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.379877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.379844 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xlqw\" (UniqueName: \"kubernetes.io/projected/6093cc86-105f-4090-bc83-b334effee41a-kube-api-access-4xlqw\") pod \"cert-manager-597b95cb84-kkx5w\" (UID: \"6093cc86-105f-4090-bc83-b334effee41a\") " pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.380018 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.379905 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6093cc86-105f-4090-bc83-b334effee41a-bound-sa-token\") pod \"cert-manager-597b95cb84-kkx5w\" (UID: \"6093cc86-105f-4090-bc83-b334effee41a\") " pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.383405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.383374 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kkx5w"] Nov 04 03:22:19.432113 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.432088 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4hksm\"" Nov 04 03:22:19.480525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.480499 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xlqw\" (UniqueName: \"kubernetes.io/projected/6093cc86-105f-4090-bc83-b334effee41a-kube-api-access-4xlqw\") pod \"cert-manager-597b95cb84-kkx5w\" (UID: \"6093cc86-105f-4090-bc83-b334effee41a\") " pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.480643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.480550 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6093cc86-105f-4090-bc83-b334effee41a-bound-sa-token\") pod \"cert-manager-597b95cb84-kkx5w\" (UID: \"6093cc86-105f-4090-bc83-b334effee41a\") " pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.543519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.543492 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xlqw\" (UniqueName: \"kubernetes.io/projected/6093cc86-105f-4090-bc83-b334effee41a-kube-api-access-4xlqw\") pod \"cert-manager-597b95cb84-kkx5w\" (UID: \"6093cc86-105f-4090-bc83-b334effee41a\") " pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.547027 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.547003 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6093cc86-105f-4090-bc83-b334effee41a-bound-sa-token\") pod \"cert-manager-597b95cb84-kkx5w\" (UID: \"6093cc86-105f-4090-bc83-b334effee41a\") " pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:19.637334 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:19.637302 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-kkx5w" Nov 04 03:22:20.007356 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:20.007206 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-kkx5w"] Nov 04 03:22:20.009868 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:22:20.009841 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6093cc86_105f_4090_bc83_b334effee41a.slice/crio-11bfcc215ee721cd560d75e2aa0b894fa65681b43d82815b5c0071d653b63867 WatchSource:0}: Error finding container 11bfcc215ee721cd560d75e2aa0b894fa65681b43d82815b5c0071d653b63867: Status 404 returned error can't find the container with id 11bfcc215ee721cd560d75e2aa0b894fa65681b43d82815b5c0071d653b63867 Nov 04 03:22:20.073170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:20.073139 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kkx5w" event={"ID":"6093cc86-105f-4090-bc83-b334effee41a","Type":"ContainerStarted","Data":"9cb7a88725973146eac960d017c80c30be08d66e0620d31c32171a2e40ed1be2"} Nov 04 03:22:20.073533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:20.073185 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-kkx5w" event={"ID":"6093cc86-105f-4090-bc83-b334effee41a","Type":"ContainerStarted","Data":"11bfcc215ee721cd560d75e2aa0b894fa65681b43d82815b5c0071d653b63867"} Nov 04 03:22:20.195857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:20.195802 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-kkx5w" podStartSLOduration=1.195786282 podStartE2EDuration="1.195786282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:20.194822554 +0000 UTC m=+317.578250585" watchObservedRunningTime="2026-11-04 03:22:20.195786282 +0000 UTC m=+317.579214323" Nov 04 03:22:20.931347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:20.931313 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:21.175882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:21.175841 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:23.087485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:23.087429 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" event={"ID":"977397c0-4ef8-4724-a10e-95e85fde7208","Type":"ContainerStarted","Data":"f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099"} Nov 04 03:22:23.087929 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:23.087662 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:23.101159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:23.101127 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:23.206792 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:23.206736 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" podStartSLOduration=2.825160479 podStartE2EDuration="6.206724124s" totalImagesPullingTime="3.381563645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:23.206634325 +0000 UTC m=+320.590062370" watchObservedRunningTime="2026-11-04 03:22:23.206724124 +0000 UTC m=+320.590152163" Nov 04 03:22:23.358406 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:23.358330 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:25.761314 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:25.761280 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:26.066959 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:26.066882 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:30.919060 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:30.919027 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:31.221539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:31.221437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:31.471794 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:31.471707 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:31.907871 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:31.907831 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:34.092767 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:34.092739 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:22:37.577674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:37.577633 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:37.875001 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:37.874925 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:42.329610 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:42.329579 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:42.571677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:42.571642 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:46.353626 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:22:46.353591 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:22:47.136590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:47.136547 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:47.438916 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:47.438823 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:50.065175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.065143 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:22:50.068910 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.068895 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.126778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.126749 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:22:50.248891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.248857 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92vz\" (UniqueName: \"kubernetes.io/projected/eab5b6b7-8940-4e9e-91d8-d3679f390a34-kube-api-access-x92vz\") pod \"limitador-limitador-7c9b87d574-24x8l\" (UID: \"eab5b6b7-8940-4e9e-91d8-d3679f390a34\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.249063 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.248901 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eab5b6b7-8940-4e9e-91d8-d3679f390a34-config-file\") pod \"limitador-limitador-7c9b87d574-24x8l\" (UID: \"eab5b6b7-8940-4e9e-91d8-d3679f390a34\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.299213 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.299181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:50.350177 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.350143 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eab5b6b7-8940-4e9e-91d8-d3679f390a34-config-file\") pod \"limitador-limitador-7c9b87d574-24x8l\" (UID: \"eab5b6b7-8940-4e9e-91d8-d3679f390a34\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.350345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.350271 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92vz\" (UniqueName: \"kubernetes.io/projected/eab5b6b7-8940-4e9e-91d8-d3679f390a34-kube-api-access-x92vz\") pod \"limitador-limitador-7c9b87d574-24x8l\" (UID: \"eab5b6b7-8940-4e9e-91d8-d3679f390a34\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.350936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.350915 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eab5b6b7-8940-4e9e-91d8-d3679f390a34-config-file\") pod \"limitador-limitador-7c9b87d574-24x8l\" (UID: \"eab5b6b7-8940-4e9e-91d8-d3679f390a34\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.385121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.385092 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:22:50.412150 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.412115 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92vz\" (UniqueName: \"kubernetes.io/projected/eab5b6b7-8940-4e9e-91d8-d3679f390a34-kube-api-access-x92vz\") pod \"limitador-limitador-7c9b87d574-24x8l\" (UID: \"eab5b6b7-8940-4e9e-91d8-d3679f390a34\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.600453 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.600360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:50.680252 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.680216 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:50.844944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:50.844906 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:51.048796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:51.048772 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:22:51.051386 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:22:51.051358 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab5b6b7_8940_4e9e_91d8_d3679f390a34.slice/crio-42247487187fbcfef4f100e30f7fc2b14fb8e24a0f5704a9b5b3d64c68628322 WatchSource:0}: Error finding container 42247487187fbcfef4f100e30f7fc2b14fb8e24a0f5704a9b5b3d64c68628322: Status 404 returned error can't find the container with id 42247487187fbcfef4f100e30f7fc2b14fb8e24a0f5704a9b5b3d64c68628322 Nov 04 03:22:51.202809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:51.202772 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" event={"ID":"eab5b6b7-8940-4e9e-91d8-d3679f390a34","Type":"ContainerStarted","Data":"33d3e55faa4cf2e57838b16ab8c291904f151ed40abc153a24ccaa2cdbd40aba"} Nov 04 03:22:51.203190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:51.202827 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" event={"ID":"eab5b6b7-8940-4e9e-91d8-d3679f390a34","Type":"ContainerStarted","Data":"42247487187fbcfef4f100e30f7fc2b14fb8e24a0f5704a9b5b3d64c68628322"} Nov 04 03:22:51.203190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:51.202898 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:22:51.321305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:51.321200 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" podStartSLOduration=1.321185274 podStartE2EDuration="1.321185274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:22:51.319578891 +0000 UTC m=+348.703006931" watchObservedRunningTime="2026-11-04 03:22:51.321185274 +0000 UTC m=+348.704613335" Nov 04 03:22:51.880203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:51.880168 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:52.123826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:52.123778 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:22:54.380288 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:22:54.380250 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:02.207916 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:02.207883 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-24x8l" Nov 04 03:23:02.377438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:02.377404 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:23:02.377675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:02.377654 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" podUID="977397c0-4ef8-4724-a10e-95e85fde7208" containerName="limitador" containerID="cri-o://f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099" gracePeriod=30 Nov 04 03:23:02.927223 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:02.927199 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:23:02.954174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:02.954123 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47\" (UniqueName: \"kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47\") pod \"977397c0-4ef8-4724-a10e-95e85fde7208\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " Nov 04 03:23:02.956321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:02.956296 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47" pod "977397c0-4ef8-4724-a10e-95e85fde7208" (UID: "977397c0-4ef8-4724-a10e-95e85fde7208"). InnerVolumeSpecName "kube-api-access-8hq47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:23:03.055043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.055006 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file\" (UniqueName: \"kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file\") pod \"977397c0-4ef8-4724-a10e-95e85fde7208\" (UID: \"977397c0-4ef8-4724-a10e-95e85fde7208\") " Nov 04 03:23:03.055213 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.055179 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hq47\" (UniqueName: \"kubernetes.io/projected/977397c0-4ef8-4724-a10e-95e85fde7208-kube-api-access-8hq47\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:03.055325 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.055300 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file" pod "977397c0-4ef8-4724-a10e-95e85fde7208" (UID: "977397c0-4ef8-4724-a10e-95e85fde7208"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:23:03.155758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.155717 4959 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/977397c0-4ef8-4724-a10e-95e85fde7208-config-file\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:03.252250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.252172 4959 generic.go:356] "Generic (PLEG): container finished" podID="977397c0-4ef8-4724-a10e-95e85fde7208" containerID="f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099" exitCode=0 Nov 04 03:23:03.252250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.252223 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" Nov 04 03:23:03.252722 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.252257 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" event={"ID":"977397c0-4ef8-4724-a10e-95e85fde7208","Type":"ContainerDied","Data":"f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099"} Nov 04 03:23:03.252722 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.252349 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wsld5" event={"ID":"977397c0-4ef8-4724-a10e-95e85fde7208","Type":"ContainerDied","Data":"21f87b7b462bfd6ec2581ee8b499062e5fc51dcfe48a166fbfdf16a8c55feb93"} Nov 04 03:23:03.252722 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.252374 4959 scope.go:122] "RemoveContainer" containerID="f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099" Nov 04 03:23:03.260647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.260631 4959 scope.go:122] "RemoveContainer" containerID="f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099" Nov 04 03:23:03.260915 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:23:03.260895 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099\": container with ID starting with f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099 not found: ID does not exist" containerID="f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099" Nov 04 03:23:03.260961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.260923 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099"} err="failed to get container status \"f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099\": rpc error: code = NotFound desc = could not find container \"f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099\": container with ID starting with f33470a37c59f5695bee2574cd1f1b15fa4a000ed2259e190eef516233415099 not found: ID does not exist" Nov 04 03:23:03.418130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.418097 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:23:03.421386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:03.421363 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wsld5"] Nov 04 03:23:04.797801 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:04.797766 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977397c0-4ef8-4724-a10e-95e85fde7208" path="/var/lib/kubelet/pods/977397c0-4ef8-4724-a10e-95e85fde7208/volumes" Nov 04 03:23:08.172057 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:08.172025 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:08.468631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:08.468543 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:08.975359 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:08.975325 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:16.353956 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:23:16.353913 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:23:17.413857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:17.413820 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:17.667426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:17.667336 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:30.009741 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.009710 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t"] Nov 04 03:23:30.015837 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.015804 4959 cpu_manager.go:414] "containerMap: removing container" podUID="977397c0-4ef8-4724-a10e-95e85fde7208" containerName="limitador" Nov 04 03:23:30.016014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.015893 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977397c0-4ef8-4724-a10e-95e85fde7208" containerName="limitador" Nov 04 03:23:30.016083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.016025 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="977397c0-4ef8-4724-a10e-95e85fde7208" containerName="limitador" Nov 04 03:23:30.019114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.019091 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071198 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071230 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071254 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzh4\" (UniqueName: \"kubernetes.io/projected/d9146acf-ce25-4bff-b1f3-04b5b112052a-kube-api-access-ptzh4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071368 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071404 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9146acf-ce25-4bff-b1f3-04b5b112052a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071424 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.071651 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.071456 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.126211 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.126179 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t"] Nov 04 03:23:30.172879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.172836 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzh4\" (UniqueName: \"kubernetes.io/projected/d9146acf-ce25-4bff-b1f3-04b5b112052a-kube-api-access-ptzh4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.173053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.172918 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.173053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.172971 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9146acf-ce25-4bff-b1f3-04b5b112052a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.173053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.173007 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.173053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.173040 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.173270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.173082 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.173270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.173103 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.174114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.174084 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.175509 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.175456 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9146acf-ce25-4bff-b1f3-04b5b112052a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.175699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.175681 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.175750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.175712 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.175786 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.175756 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.234595 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.234562 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9146acf-ce25-4bff-b1f3-04b5b112052a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.235930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.235907 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzh4\" (UniqueName: \"kubernetes.io/projected/d9146acf-ce25-4bff-b1f3-04b5b112052a-kube-api-access-ptzh4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7jh5t\" (UID: \"d9146acf-ce25-4bff-b1f3-04b5b112052a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.330084 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.330056 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:30.757784 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.757755 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t"] Nov 04 03:23:30.759880 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:23:30.759851 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9146acf_ce25_4bff_b1f3_04b5b112052a.slice/crio-b356b90def13e7b570adac1285a855a31213258a6dec349ba1251b4662adb638 WatchSource:0}: Error finding container b356b90def13e7b570adac1285a855a31213258a6dec349ba1251b4662adb638: Status 404 returned error can't find the container with id b356b90def13e7b570adac1285a855a31213258a6dec349ba1251b4662adb638 Nov 04 03:23:30.761896 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.761867 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:23:30.761986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:30.761937 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:23:31.369271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:31.369228 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" event={"ID":"d9146acf-ce25-4bff-b1f3-04b5b112052a","Type":"ContainerStarted","Data":"7d17e3f2588f446015198d70492fe936f6e6365af403a336c6fe927339a37657"} Nov 04 03:23:31.369804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:31.369286 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" event={"ID":"d9146acf-ce25-4bff-b1f3-04b5b112052a","Type":"ContainerStarted","Data":"b356b90def13e7b570adac1285a855a31213258a6dec349ba1251b4662adb638"} Nov 04 03:23:31.369804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:31.369312 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:31.371087 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:31.371063 4959 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7jh5t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:23:31.371197 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:31.371108 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" podUID="d9146acf-ce25-4bff-b1f3-04b5b112052a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:23:31.492519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:31.492451 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" podStartSLOduration=2.49243758 podStartE2EDuration="2.49243758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:31.490946153 +0000 UTC m=+388.874374206" watchObservedRunningTime="2026-11-04 03:23:31.49243758 +0000 UTC m=+388.875865620" Nov 04 03:23:32.374232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.374206 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7jh5t" Nov 04 03:23:32.600175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.600141 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz"] Nov 04 03:23:32.600398 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.600373 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" podUID="9e61f3eb-4169-48e2-8c70-7969754480d6" containerName="discovery" containerID="cri-o://2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195" gracePeriod=30 Nov 04 03:23:32.846144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.846122 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:23:32.895238 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895208 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.895367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895263 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.895367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895288 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.895367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895307 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.895367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895353 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.895611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895392 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.895611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895418 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap\") pod \"9e61f3eb-4169-48e2-8c70-7969754480d6\" (UID: \"9e61f3eb-4169-48e2-8c70-7969754480d6\") " Nov 04 03:23:32.896003 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.895972 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:23:32.898289 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.898256 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:23:32.898412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.898295 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:23:32.898412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.898368 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:23:32.898412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.898372 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:23:32.898674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.898650 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:23:32.898729 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.898705 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz" pod "9e61f3eb-4169-48e2-8c70-7969754480d6" (UID: "9e61f3eb-4169-48e2-8c70-7969754480d6"). InnerVolumeSpecName "kube-api-access-d8glz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:23:32.997193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997108 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8glz\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-kube-api-access-d8glz\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:32.997193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997142 4959 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e61f3eb-4169-48e2-8c70-7969754480d6-local-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:32.997193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997158 4959 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-ca-configmap\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:32.997193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997171 4959 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-cacerts\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:32.997193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997185 4959 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-kubeconfig\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:32.997444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997198 4959 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-csr-dns-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:32.997444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:32.997213 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e61f3eb-4169-48e2-8c70-7969754480d6-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:23:33.378791 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.378752 4959 generic.go:356] "Generic (PLEG): container finished" podID="9e61f3eb-4169-48e2-8c70-7969754480d6" containerID="2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195" exitCode=0 Nov 04 03:23:33.379288 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.378803 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" Nov 04 03:23:33.379288 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.378842 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" event={"ID":"9e61f3eb-4169-48e2-8c70-7969754480d6","Type":"ContainerDied","Data":"2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195"} Nov 04 03:23:33.379288 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.378903 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz" event={"ID":"9e61f3eb-4169-48e2-8c70-7969754480d6","Type":"ContainerDied","Data":"71b95b5450f8bb79151bd08cde30e8e45895fd05176eae47a12856a0cbaa2334"} Nov 04 03:23:33.379288 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.378933 4959 scope.go:122] "RemoveContainer" containerID="2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195" Nov 04 03:23:33.387935 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.387913 4959 scope.go:122] "RemoveContainer" containerID="2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195" Nov 04 03:23:33.388186 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:23:33.388169 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195\": container with ID starting with 2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195 not found: ID does not exist" containerID="2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195" Nov 04 03:23:33.388236 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.388194 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195"} err="failed to get container status \"2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195\": rpc error: code = NotFound desc = could not find container \"2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195\": container with ID starting with 2be62f59dd3f0c38b1575abf2e1ebd694009f2cbcfb4ff669cb3bc97d63dd195 not found: ID does not exist" Nov 04 03:23:33.545803 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.545769 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz"] Nov 04 03:23:33.596334 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:33.596303 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-f4mfz"] Nov 04 03:23:34.798030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:34.797995 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e61f3eb-4169-48e2-8c70-7969754480d6" path="/var/lib/kubelet/pods/9e61f3eb-4169-48e2-8c70-7969754480d6/volumes" Nov 04 03:23:34.930932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:34.930905 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:35.251239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.251198 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:35.505982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.505897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:35.751676 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.751646 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zt9qt"] Nov 04 03:23:35.751964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.751951 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9e61f3eb-4169-48e2-8c70-7969754480d6" containerName="discovery" Nov 04 03:23:35.752008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.751966 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e61f3eb-4169-48e2-8c70-7969754480d6" containerName="discovery" Nov 04 03:23:35.752045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.752036 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e61f3eb-4169-48e2-8c70-7969754480d6" containerName="discovery" Nov 04 03:23:35.756218 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.756177 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:35.762025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.762001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:35.819509 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.819463 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t7sf\" (UniqueName: \"kubernetes.io/projected/6f9e914b-c63a-409b-9b2f-0c494ef2620b-kube-api-access-5t7sf\") pod \"kserve-controller-manager-776cc66d47-zt9qt\" (UID: \"6f9e914b-c63a-409b-9b2f-0c494ef2620b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:35.819865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.819547 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f9e914b-c63a-409b-9b2f-0c494ef2620b-cert\") pod \"kserve-controller-manager-776cc66d47-zt9qt\" (UID: \"6f9e914b-c63a-409b-9b2f-0c494ef2620b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:35.862891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.862862 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 04 03:23:35.862891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.862879 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 04 03:23:35.863079 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.862875 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l4crg\"" Nov 04 03:23:35.863079 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.862929 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 04 03:23:35.864178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.864158 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zt9qt"] Nov 04 03:23:35.921080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.921046 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t7sf\" (UniqueName: \"kubernetes.io/projected/6f9e914b-c63a-409b-9b2f-0c494ef2620b-kube-api-access-5t7sf\") pod \"kserve-controller-manager-776cc66d47-zt9qt\" (UID: \"6f9e914b-c63a-409b-9b2f-0c494ef2620b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:35.921080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.921082 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f9e914b-c63a-409b-9b2f-0c494ef2620b-cert\") pod \"kserve-controller-manager-776cc66d47-zt9qt\" (UID: \"6f9e914b-c63a-409b-9b2f-0c494ef2620b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:35.924431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.924404 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f9e914b-c63a-409b-9b2f-0c494ef2620b-cert\") pod \"kserve-controller-manager-776cc66d47-zt9qt\" (UID: \"6f9e914b-c63a-409b-9b2f-0c494ef2620b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:35.986592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:35.986556 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t7sf\" (UniqueName: \"kubernetes.io/projected/6f9e914b-c63a-409b-9b2f-0c494ef2620b-kube-api-access-5t7sf\") pod \"kserve-controller-manager-776cc66d47-zt9qt\" (UID: \"6f9e914b-c63a-409b-9b2f-0c494ef2620b\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:36.013485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.013394 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:36.068482 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.068450 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:36.096764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.096735 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zm64c"] Nov 04 03:23:36.100747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.100728 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.163297 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.163259 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 04 03:23:36.163502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.163335 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pmc7d\"" Nov 04 03:23:36.163502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.163352 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 04 03:23:36.164145 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.164123 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zm64c"] Nov 04 03:23:36.224776 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.224735 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97hb\" (UniqueName: \"kubernetes.io/projected/8bfb06a1-e89e-4063-bf27-e580ae21a09b-kube-api-access-r97hb\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.224776 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.224784 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfb06a1-e89e-4063-bf27-e580ae21a09b-metrics-certs\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.224990 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.224807 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bfb06a1-e89e-4063-bf27-e580ae21a09b-cert\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.325243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.325215 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97hb\" (UniqueName: \"kubernetes.io/projected/8bfb06a1-e89e-4063-bf27-e580ae21a09b-kube-api-access-r97hb\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.325377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.325253 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfb06a1-e89e-4063-bf27-e580ae21a09b-metrics-certs\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.325377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.325279 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bfb06a1-e89e-4063-bf27-e580ae21a09b-cert\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.327958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.327932 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bfb06a1-e89e-4063-bf27-e580ae21a09b-cert\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.328067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.327974 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfb06a1-e89e-4063-bf27-e580ae21a09b-metrics-certs\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.386858 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.386825 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97hb\" (UniqueName: \"kubernetes.io/projected/8bfb06a1-e89e-4063-bf27-e580ae21a09b-kube-api-access-r97hb\") pod \"odh-model-controller-769875db56-zm64c\" (UID: \"8bfb06a1-e89e-4063-bf27-e580ae21a09b\") " pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.414842 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.414813 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:36.438418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.438390 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zt9qt"] Nov 04 03:23:36.441587 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:23:36.441556 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9e914b_c63a_409b_9b2f_0c494ef2620b.slice/crio-2d930b19404238e98891735e47aefc64c6e95344b92aee929882a42492ab16c8 WatchSource:0}: Error finding container 2d930b19404238e98891735e47aefc64c6e95344b92aee929882a42492ab16c8: Status 404 returned error can't find the container with id 2d930b19404238e98891735e47aefc64c6e95344b92aee929882a42492ab16c8 Nov 04 03:23:36.784718 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:36.784691 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zm64c"] Nov 04 03:23:36.787343 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:23:36.787317 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfb06a1_e89e_4063_bf27_e580ae21a09b.slice/crio-ebe950033e1c40314e441fe8370e3a6577981df491a1068fb089b4084d079e18 WatchSource:0}: Error finding container ebe950033e1c40314e441fe8370e3a6577981df491a1068fb089b4084d079e18: Status 404 returned error can't find the container with id ebe950033e1c40314e441fe8370e3a6577981df491a1068fb089b4084d079e18 Nov 04 03:23:37.206949 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:37.206910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:37.399952 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:37.399920 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" event={"ID":"6f9e914b-c63a-409b-9b2f-0c494ef2620b","Type":"ContainerStarted","Data":"2d930b19404238e98891735e47aefc64c6e95344b92aee929882a42492ab16c8"} Nov 04 03:23:37.400996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:37.400966 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zm64c" event={"ID":"8bfb06a1-e89e-4063-bf27-e580ae21a09b","Type":"ContainerStarted","Data":"ebe950033e1c40314e441fe8370e3a6577981df491a1068fb089b4084d079e18"} Nov 04 03:23:40.419811 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:40.419724 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" event={"ID":"6f9e914b-c63a-409b-9b2f-0c494ef2620b","Type":"ContainerStarted","Data":"0d629ca39526e1fb8a3f40f139557bab0ed30c1199a0f4bc8b82a2754b0240d9"} Nov 04 03:23:40.420247 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:40.419926 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:23:40.421301 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:40.421279 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zm64c" event={"ID":"8bfb06a1-e89e-4063-bf27-e580ae21a09b","Type":"ContainerStarted","Data":"9a8f3a01039f150e51e0e3594a7b74f09a40d14ffe1b8bb74f504b7ebb202494"} Nov 04 03:23:40.421412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:40.421398 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:40.538034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:40.537969 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" podStartSLOduration=2.079053813 podStartE2EDuration="5.53795627s" totalImagesPullingTime="3.458902457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:40.537121288 +0000 UTC m=+397.920549331" watchObservedRunningTime="2026-11-04 03:23:40.53795627 +0000 UTC m=+397.921384334" Nov 04 03:23:40.657424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:40.657368 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zm64c" podStartSLOduration=1.30163384 podStartE2EDuration="4.657352071s" totalImagesPullingTime="3.355718231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:40.656289985 +0000 UTC m=+398.039718026" watchObservedRunningTime="2026-11-04 03:23:40.657352071 +0000 UTC m=+398.040780166" Nov 04 03:23:42.356257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:42.356159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:42.661020 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:42.660939 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:42.911441 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:42.911342 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:43.160926 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:43.160895 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:43.406528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:43.406489 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:43.679000 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:43.678890 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:46.354144 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:23:46.354108 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:23:50.021904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.021865 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8456fc546f-85nz4"] Nov 04 03:23:50.025282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.025264 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.063503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.063447 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:50.078257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.078231 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 04 03:23:50.078412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.078368 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 04 03:23:50.128891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.128853 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dfrbx\"" Nov 04 03:23:50.128891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.128854 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 04 03:23:50.130317 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.130295 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8456fc546f-85nz4"] Nov 04 03:23:50.146150 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.146123 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.146230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.146178 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.146314 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.146295 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78lq\" (UniqueName: \"kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.247246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.247216 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.247435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.247271 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78lq\" (UniqueName: \"kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.247435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.247324 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.249915 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.249890 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.250005 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.249950 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.313881 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.313809 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78lq\" (UniqueName: \"kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq\") pod \"maas-controller-8456fc546f-85nz4\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.337723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.337690 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:50.358250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.358201 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:50.610410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.610381 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:50.708240 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:50.708217 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8456fc546f-85nz4"] Nov 04 03:23:50.710873 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:23:50.710849 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58c8d0c_36a4_4c15_bae8_4c0adccf25ce.slice/crio-53da48dddb32ffbb1c2a2ea0f1c03a590499d66a2a9db6c4f771a2aa77d4432f WatchSource:0}: Error finding container 53da48dddb32ffbb1c2a2ea0f1c03a590499d66a2a9db6c4f771a2aa77d4432f: Status 404 returned error can't find the container with id 53da48dddb32ffbb1c2a2ea0f1c03a590499d66a2a9db6c4f771a2aa77d4432f Nov 04 03:23:51.427803 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:51.427771 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zm64c" Nov 04 03:23:51.466849 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:51.466813 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8456fc546f-85nz4" event={"ID":"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce","Type":"ContainerStarted","Data":"53da48dddb32ffbb1c2a2ea0f1c03a590499d66a2a9db6c4f771a2aa77d4432f"} Nov 04 03:23:54.731338 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:54.731302 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:59.508890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:59.508804 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8456fc546f-85nz4" event={"ID":"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce","Type":"ContainerStarted","Data":"0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05"} Nov 04 03:23:59.509244 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:59.508956 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:23:59.566804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:59.566767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:23:59.628277 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:59.628221 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8456fc546f-85nz4" podStartSLOduration=2.138304093 podStartE2EDuration="10.628207804s" totalImagesPullingTime="8.489903711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:23:59.627441996 +0000 UTC m=+417.010870036" watchObservedRunningTime="2026-11-04 03:23:59.628207804 +0000 UTC m=+417.011635841" Nov 04 03:23:59.813869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:23:59.813774 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:03.062871 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:03.062833 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:03.309875 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:03.309841 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:08.131287 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:08.131228 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:08.378985 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:08.378954 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:09.470560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.470521 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp"] Nov 04 03:24:09.474045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.474019 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.513392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.513354 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb6q8\" (UniqueName: \"kubernetes.io/projected/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-kube-api-access-pb6q8\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.513392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.513406 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-manager-config\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.513664 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.513515 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-cert\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.513664 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.513552 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-metrics-certs\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.577756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.577722 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 04 03:24:09.577919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.577765 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 04 03:24:09.577919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.577775 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 04 03:24:09.577919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.577856 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 04 03:24:09.577919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.577903 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-hhzn9\"" Nov 04 03:24:09.579263 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.579246 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp"] Nov 04 03:24:09.614406 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.614369 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb6q8\" (UniqueName: \"kubernetes.io/projected/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-kube-api-access-pb6q8\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.614585 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.614422 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-manager-config\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.614585 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.614449 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-cert\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.614691 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.614585 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-metrics-certs\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.615160 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.615135 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-manager-config\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.617133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.617113 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-metrics-certs\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.628602 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.628580 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 04 03:24:09.637957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.637931 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-cert\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.676990 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.676954 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb6q8\" (UniqueName: \"kubernetes.io/projected/d6abdaa5-0100-4c16-b2f6-3d56d8ec8129-kube-api-access-pb6q8\") pod \"jobset-controller-manager-7578997f76-wwszp\" (UID: \"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129\") " pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:09.785276 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:09.785199 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:10.160209 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.160182 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp"] Nov 04 03:24:10.162280 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:24:10.162252 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6abdaa5_0100_4c16_b2f6_3d56d8ec8129.slice/crio-01fb3c6e4096a721cf0041440b40d456cbe2d408ecf0e2f0b413bcc1e98a38f4 WatchSource:0}: Error finding container 01fb3c6e4096a721cf0041440b40d456cbe2d408ecf0e2f0b413bcc1e98a38f4: Status 404 returned error can't find the container with id 01fb3c6e4096a721cf0041440b40d456cbe2d408ecf0e2f0b413bcc1e98a38f4 Nov 04 03:24:10.515428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.515356 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:24:10.560772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.560743 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" event={"ID":"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129","Type":"ContainerStarted","Data":"01fb3c6e4096a721cf0041440b40d456cbe2d408ecf0e2f0b413bcc1e98a38f4"} Nov 04 03:24:10.626209 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.626186 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8456fc546f-85nz4"] Nov 04 03:24:10.626352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.626334 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-8456fc546f-85nz4" podUID="e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" containerName="manager" containerID="cri-o://0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05" gracePeriod=30 Nov 04 03:24:10.858881 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.858846 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:24:10.926403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.926374 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs\") pod \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " Nov 04 03:24:10.926553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.926418 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert\") pod \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " Nov 04 03:24:10.926553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.926458 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq\" (UniqueName: \"kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq\") pod \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\" (UID: \"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce\") " Nov 04 03:24:10.928914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.928881 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs" pod "e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" (UID: "e58c8d0c-36a4-4c15-bae8-4c0adccf25ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:24:10.929021 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.928910 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq" pod "e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" (UID: "e58c8d0c-36a4-4c15-bae8-4c0adccf25ce"). InnerVolumeSpecName "kube-api-access-s78lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:24:10.929021 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:10.928947 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert" pod "e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" (UID: "e58c8d0c-36a4-4c15-bae8-4c0adccf25ce"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:24:11.027766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.027736 4959 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-metrics-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:11.027766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.027760 4959 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-webhook-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:11.027766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.027771 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78lq\" (UniqueName: \"kubernetes.io/projected/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce-kube-api-access-s78lq\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:24:11.431490 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.431450 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-zt9qt" Nov 04 03:24:11.567082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.567043 4959 generic.go:356] "Generic (PLEG): container finished" podID="e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" containerID="0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05" exitCode=0 Nov 04 03:24:11.567591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.567106 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8456fc546f-85nz4" event={"ID":"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce","Type":"ContainerDied","Data":"0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05"} Nov 04 03:24:11.567591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.567172 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8456fc546f-85nz4" Nov 04 03:24:11.567591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.567202 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8456fc546f-85nz4" event={"ID":"e58c8d0c-36a4-4c15-bae8-4c0adccf25ce","Type":"ContainerDied","Data":"53da48dddb32ffbb1c2a2ea0f1c03a590499d66a2a9db6c4f771a2aa77d4432f"} Nov 04 03:24:11.567591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.567228 4959 scope.go:122] "RemoveContainer" containerID="0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05" Nov 04 03:24:11.576384 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.576366 4959 scope.go:122] "RemoveContainer" containerID="0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05" Nov 04 03:24:11.576789 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:24:11.576768 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05\": container with ID starting with 0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05 not found: ID does not exist" containerID="0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05" Nov 04 03:24:11.576866 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.576807 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05"} err="failed to get container status \"0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05\": rpc error: code = NotFound desc = could not find container \"0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05\": container with ID starting with 0ca5990a008f2253c2412a894778e2de37958011499ee9e220fc3f5c8f69ce05 not found: ID does not exist" Nov 04 03:24:11.733847 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.733755 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8456fc546f-85nz4"] Nov 04 03:24:11.741011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:11.740975 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8456fc546f-85nz4"] Nov 04 03:24:12.799929 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:12.799891 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" path="/var/lib/kubelet/pods/e58c8d0c-36a4-4c15-bae8-4c0adccf25ce/volumes" Nov 04 03:24:14.585149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:14.585114 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" event={"ID":"d6abdaa5-0100-4c16-b2f6-3d56d8ec8129","Type":"ContainerStarted","Data":"097bb51c008ee3cf4885fea33bde83002165a9f3beab6c71b9c1984972c14916"} Nov 04 03:24:14.585549 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:14.585221 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:14.708699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:14.708646 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" podStartSLOduration=1.728111954 podStartE2EDuration="5.708631653s" totalImagesPullingTime="3.980519699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:24:14.707319433 +0000 UTC m=+432.090747473" watchObservedRunningTime="2026-11-04 03:24:14.708631653 +0000 UTC m=+432.092059722" Nov 04 03:24:16.354494 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:24:16.354438 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:24:25.595235 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:25.595205 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7578997f76-wwszp" Nov 04 03:24:33.266809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:33.266776 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:33.567734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:33.567638 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:35.379350 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:35.379307 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:35.677192 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:35.677098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:35.929718 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:35.929629 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:39.937946 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:39.937903 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:40.189512 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:40.189388 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:41.855921 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:41.855890 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:24:46.355085 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:24:46.355050 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:24:51.413336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:51.413301 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:51.669933 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:51.669848 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:24:55.077520 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:24:55.077483 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:16.353547 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:25:16.353443 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:25:28.734857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:28.734808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:28.982284 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:28.982247 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:31.910263 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:31.910227 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:32.156504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:32.156455 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:37.424405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:37.424359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:37.676864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:37.676772 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:42.370229 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:42.370188 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:42.620702 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:42.620608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:46.354140 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:25:46.354104 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:25:55.430868 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:55.430828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:57.981354 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:57.981323 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:58.230986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:58.230952 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:25:59.940544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:25:59.940509 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:00.915507 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:00.915455 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:01.405959 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:01.405923 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:02.637723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:02.637686 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:02.960895 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:02.960826 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:06.933733 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:06.933692 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:07.177281 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:07.177244 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:09.344127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:09.344094 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:13.032904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:13.032865 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:26:16.354592 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:26:16.354555 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:26:17.962626 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:17.962593 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:20.515463 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:20.515432 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:26:21.912008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:21.911975 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:24.939788 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:24.939757 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp"] Nov 04 03:26:24.940176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:24.940074 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" containerName="manager" Nov 04 03:26:24.940176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:24.940084 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" containerName="manager" Nov 04 03:26:24.940176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:24.940154 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58c8d0c-36a4-4c15-bae8-4c0adccf25ce" containerName="manager" Nov 04 03:26:24.942985 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:24.942967 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:24.995996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:24.995965 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 04 03:26:25.046444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.046409 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 04 03:26:25.046655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.046529 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 04 03:26:25.046655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.046529 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8fl6l\"" Nov 04 03:26:25.047758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.047740 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp"] Nov 04 03:26:25.074790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.074760 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf9e67f-87bf-43da-a257-d8a2d6171613-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.074936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.074800 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.074936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.074820 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.074936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.074901 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.075106 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.075033 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfrf\" (UniqueName: \"kubernetes.io/projected/dbf9e67f-87bf-43da-a257-d8a2d6171613-kube-api-access-wvfrf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.075106 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.075072 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.149239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.149197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:25.176345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176314 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfrf\" (UniqueName: \"kubernetes.io/projected/dbf9e67f-87bf-43da-a257-d8a2d6171613-kube-api-access-wvfrf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176345 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176391 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf9e67f-87bf-43da-a257-d8a2d6171613-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176432 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176492 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176534 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176823 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176800 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176880 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176855 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.176921 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.176909 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.178842 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.178823 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbf9e67f-87bf-43da-a257-d8a2d6171613-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.179149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.179131 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbf9e67f-87bf-43da-a257-d8a2d6171613-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.237928 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.237859 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfrf\" (UniqueName: \"kubernetes.io/projected/dbf9e67f-87bf-43da-a257-d8a2d6171613-kube-api-access-wvfrf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pqqxp\" (UID: \"dbf9e67f-87bf-43da-a257-d8a2d6171613\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.255720 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.255690 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:25.633835 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:25.633808 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp"] Nov 04 03:26:25.636791 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:26:25.636762 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf9e67f_87bf_43da_a257_d8a2d6171613.slice/crio-0470aa508c4885ce01934e7c2a20f337af0034716591ca13fe69e35a8572e960 WatchSource:0}: Error finding container 0470aa508c4885ce01934e7c2a20f337af0034716591ca13fe69e35a8572e960: Status 404 returned error can't find the container with id 0470aa508c4885ce01934e7c2a20f337af0034716591ca13fe69e35a8572e960 Nov 04 03:26:26.125154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:26.125119 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" event={"ID":"dbf9e67f-87bf-43da-a257-d8a2d6171613","Type":"ContainerStarted","Data":"0470aa508c4885ce01934e7c2a20f337af0034716591ca13fe69e35a8572e960"} Nov 04 03:26:28.135182 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:28.135142 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" event={"ID":"dbf9e67f-87bf-43da-a257-d8a2d6171613","Type":"ContainerStarted","Data":"6fb3143053ebc1cddfea232d9d896656e4ead8b94aa87f36a214fa887718acc7"} Nov 04 03:26:28.254007 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:28.253956 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" podStartSLOduration=2.741417727 podStartE2EDuration="4.25394277s" totalImagesPullingTime="1.512525043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:28.252368194 +0000 UTC m=+565.635796245" watchObservedRunningTime="2026-11-04 03:26:28.25394277 +0000 UTC m=+565.637370810" Nov 04 03:26:28.255777 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:28.255759 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:29.093630 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:29.093595 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:26:32.101057 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:32.101028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:35.256082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.256042 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:35.268723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.268696 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:35.438892 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.438860 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zg7ps"] Nov 04 03:26:35.442216 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.442195 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.495298 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.495268 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 04 03:26:35.502593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.502563 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zg7ps"] Nov 04 03:26:35.542385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.542287 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:35.567542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.567514 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-home\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.567542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.567546 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.567754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.567567 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.567754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.567649 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.567819 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.567751 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0057f972-ac5a-420a-856d-f5cbac9d7622-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.567852 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.567818 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xxf\" (UniqueName: \"kubernetes.io/projected/0057f972-ac5a-420a-856d-f5cbac9d7622-kube-api-access-25xxf\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.668628 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.668586 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0057f972-ac5a-420a-856d-f5cbac9d7622-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.668814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.668688 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xxf\" (UniqueName: \"kubernetes.io/projected/0057f972-ac5a-420a-856d-f5cbac9d7622-kube-api-access-25xxf\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.668814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.668742 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-home\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.668895 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.668858 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.668945 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.668896 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.669072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.669052 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.669169 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.669143 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-home\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.669282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.669262 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.669396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.669375 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.671152 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.671132 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0057f972-ac5a-420a-856d-f5cbac9d7622-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.671421 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.671405 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0057f972-ac5a-420a-856d-f5cbac9d7622-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.730773 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.730740 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xxf\" (UniqueName: \"kubernetes.io/projected/0057f972-ac5a-420a-856d-f5cbac9d7622-kube-api-access-25xxf\") pod \"echo-simulator-kserve-64b6f8f96c-zg7ps\" (UID: \"0057f972-ac5a-420a-856d-f5cbac9d7622\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:35.755095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:35.755069 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:36.129176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:36.129048 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zg7ps"] Nov 04 03:26:36.131545 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:26:36.131518 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0057f972_ac5a_420a_856d_f5cbac9d7622.slice/crio-ff13e09d8afdfa23af2bcc14365ea3514cc5774832a03fee5aacf4c9879809b6 WatchSource:0}: Error finding container ff13e09d8afdfa23af2bcc14365ea3514cc5774832a03fee5aacf4c9879809b6: Status 404 returned error can't find the container with id ff13e09d8afdfa23af2bcc14365ea3514cc5774832a03fee5aacf4c9879809b6 Nov 04 03:26:36.172408 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:36.172373 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" event={"ID":"0057f972-ac5a-420a-856d-f5cbac9d7622","Type":"ContainerStarted","Data":"ff13e09d8afdfa23af2bcc14365ea3514cc5774832a03fee5aacf4c9879809b6"} Nov 04 03:26:36.184147 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:36.184120 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pqqxp" Nov 04 03:26:38.934320 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:38.934243 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:39.187355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:39.187270 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" event={"ID":"0057f972-ac5a-420a-856d-f5cbac9d7622","Type":"ContainerStarted","Data":"4734a549a03eda3c03de2c57d1c3aeff6357a145915d66bc031a5e66023d6169"} Nov 04 03:26:39.307405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:39.307352 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" podStartSLOduration=1.7968107020000001 podStartE2EDuration="4.30733698s" totalImagesPullingTime="2.510526278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:39.305947911 +0000 UTC m=+576.689375973" watchObservedRunningTime="2026-11-04 03:26:39.30733698 +0000 UTC m=+576.690765020" Nov 04 03:26:39.755334 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:39.755300 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:42.714687 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.714602 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5"] Nov 04 03:26:42.718211 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.718188 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.724142 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.724111 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.724281 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.724194 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrgww\" (UniqueName: \"kubernetes.io/projected/72981a99-2a4a-4b23-aaf3-84733ccb53b5-kube-api-access-hrgww\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.724281 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.724225 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.724635 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.724429 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.724635 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.724464 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72981a99-2a4a-4b23-aaf3-84733ccb53b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.724635 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.724617 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.821276 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.821211 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 04 03:26:42.822940 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.822906 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5"] Nov 04 03:26:42.825298 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825274 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825314 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825354 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrgww\" (UniqueName: \"kubernetes.io/projected/72981a99-2a4a-4b23-aaf3-84733ccb53b5-kube-api-access-hrgww\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825389 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825602 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825576 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825612 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72981a99-2a4a-4b23-aaf3-84733ccb53b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825759 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825716 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.825810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.825764 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.826095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.826076 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.828016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.827993 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72981a99-2a4a-4b23-aaf3-84733ccb53b5-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.828196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.828179 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72981a99-2a4a-4b23-aaf3-84733ccb53b5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.891228 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.891186 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrgww\" (UniqueName: \"kubernetes.io/projected/72981a99-2a4a-4b23-aaf3-84733ccb53b5-kube-api-access-hrgww\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5\" (UID: \"72981a99-2a4a-4b23-aaf3-84733ccb53b5\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:42.907080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.907049 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:26:42.910286 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:42.910261 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:43.029042 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:43.028964 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:43.407156 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:43.407113 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5"] Nov 04 03:26:43.409328 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:26:43.409301 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72981a99_2a4a_4b23_aaf3_84733ccb53b5.slice/crio-45902d36cca4704d1caaa48ca7e1a2f6f6b70b74e5a5fdcc2df9d3e160f8e5b4 WatchSource:0}: Error finding container 45902d36cca4704d1caaa48ca7e1a2f6f6b70b74e5a5fdcc2df9d3e160f8e5b4: Status 404 returned error can't find the container with id 45902d36cca4704d1caaa48ca7e1a2f6f6b70b74e5a5fdcc2df9d3e160f8e5b4 Nov 04 03:26:44.209067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:44.209021 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" event={"ID":"72981a99-2a4a-4b23-aaf3-84733ccb53b5","Type":"ContainerStarted","Data":"a589d8d718c1274a2e40bd8c51aaccf213023a74215d49234d8729a9988d30a9"} Nov 04 03:26:44.209067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:44.209066 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" event={"ID":"72981a99-2a4a-4b23-aaf3-84733ccb53b5","Type":"ContainerStarted","Data":"45902d36cca4704d1caaa48ca7e1a2f6f6b70b74e5a5fdcc2df9d3e160f8e5b4"} Nov 04 03:26:44.327925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:44.327873 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" podStartSLOduration=1.7379777650000001 podStartE2EDuration="2.327857449s" totalImagesPullingTime="589.879684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:44.32716515 +0000 UTC m=+581.710593224" watchObservedRunningTime="2026-11-04 03:26:44.327857449 +0000 UTC m=+581.711285489" Nov 04 03:26:45.029185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:45.029148 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:45.755653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:45.755618 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:45.768205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:45.768176 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:46.228994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:46.228967 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zg7ps" Nov 04 03:26:46.354862 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:26:46.354821 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:26:48.318616 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.318588 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9"] Nov 04 03:26:48.321957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.321940 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.365962 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.365928 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.365962 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.365962 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.365981 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.366117 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.366170 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.366227 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.366270 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.366294 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blg8x\" (UniqueName: \"kubernetes.io/projected/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-kube-api-access-blg8x\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.366339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.366315 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.374654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.374622 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cr7gn\"" Nov 04 03:26:48.466922 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.466887 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.466942 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.466975 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467059 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467104 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blg8x\" (UniqueName: \"kubernetes.io/projected/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-kube-api-access-blg8x\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467140 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467222 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467245 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467276 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467391 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467421 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467489 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.467763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.467693 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.468130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.468107 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.469774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.469746 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.469918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.469900 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.521929 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.521894 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9"] Nov 04 03:26:48.529040 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.529006 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.529171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.529077 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blg8x\" (UniqueName: \"kubernetes.io/projected/d86f1fc3-dd20-4598-a5d0-03c40cbc15c5-kube-api-access-blg8x\") pod \"maas-default-gateway-openshift-default-6dcf757899-5s8q9\" (UID: \"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:48.634006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:48.633965 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:49.059423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:49.059344 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9"] Nov 04 03:26:49.061686 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:26:49.061659 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86f1fc3_dd20_4598_a5d0_03c40cbc15c5.slice/crio-06364d867662025a0fbf44e2eed65e46b9d57575bdffca296f4fc581d15e34c8 WatchSource:0}: Error finding container 06364d867662025a0fbf44e2eed65e46b9d57575bdffca296f4fc581d15e34c8: Status 404 returned error can't find the container with id 06364d867662025a0fbf44e2eed65e46b9d57575bdffca296f4fc581d15e34c8 Nov 04 03:26:49.231293 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:49.231261 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" event={"ID":"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5","Type":"ContainerStarted","Data":"06364d867662025a0fbf44e2eed65e46b9d57575bdffca296f4fc581d15e34c8"} Nov 04 03:26:52.739656 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:52.739626 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:26:52.739927 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:52.739701 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:26:52.739927 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:52.739732 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:26:53.029122 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.029045 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:53.044028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.044000 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:53.254553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.254516 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" event={"ID":"d86f1fc3-dd20-4598-a5d0-03c40cbc15c5","Type":"ContainerStarted","Data":"047cc0d5cf91221da5299ff5b1eefb3b620e68e529e3eb2068d30d796ff49819"} Nov 04 03:26:53.267053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.267020 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5" Nov 04 03:26:53.378276 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.378223 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podStartSLOduration=1.7030399699999998 podStartE2EDuration="5.378208093s" totalImagesPullingTime="3.675168123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:26:53.37679074 +0000 UTC m=+590.760218803" watchObservedRunningTime="2026-11-04 03:26:53.378208093 +0000 UTC m=+590.761636133" Nov 04 03:26:53.634465 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.634375 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:26:53.636385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.636352 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:53.636557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:53.636438 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:26:54.635300 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:54.635266 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:54.635750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:54.635321 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:26:55.635350 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:55.635316 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:55.635788 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:55.635376 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:26:55.779734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:55.779699 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:26:56.165040 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:56.165006 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:26:56.635042 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:56.635006 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:56.635223 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:56.635069 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:26:57.635232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:57.635198 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:57.635700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:57.635247 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:26:58.634916 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:58.634872 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:58.635102 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:58.634927 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:26:59.635068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:59.634994 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5s8q9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 04 03:26:59.635634 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:26:59.635116 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" podUID="d86f1fc3-dd20-4598-a5d0-03c40cbc15c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 04 03:27:01.213936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:01.213904 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:27:01.214377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:01.214357 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:27:01.215159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:01.215138 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5s8q9" Nov 04 03:27:02.829900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:02.829868 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:27:02.830360 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:02.829969 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:27:02.836283 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:02.836262 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:27:02.836408 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:02.836347 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:27:03.446961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.446926 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh"] Nov 04 03:27:03.451760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.451743 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.494987 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.494952 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495011 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495083 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495130 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495147 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495267 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495305 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495322 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkdm\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.495403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.495379 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.551256 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.551225 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh"] Nov 04 03:27:03.596399 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596368 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596422 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596440 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596488 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596503 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596527 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596581 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596623 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkdm\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596666 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596827 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.596925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.596884 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.597247 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.597227 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.597323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.597229 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.597323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.597300 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.599169 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.599141 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.599266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.599232 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.658253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.658220 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.658525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.658502 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkdm\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm\") pod \"maas-default-gateway-openshift-default-6dcf757899-skdkh\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:03.764050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:03.763963 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:04.192359 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.192250 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh"] Nov 04 03:27:04.194525 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:27:04.194494 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb05ae7_e6f0_4784_883b_ea2d70d40dd7.slice/crio-18b3eae9f891b1f7fad40142978845ff8f6597aa2d7387d1426d62d268a5f8cf WatchSource:0}: Error finding container 18b3eae9f891b1f7fad40142978845ff8f6597aa2d7387d1426d62d268a5f8cf: Status 404 returned error can't find the container with id 18b3eae9f891b1f7fad40142978845ff8f6597aa2d7387d1426d62d268a5f8cf Nov 04 03:27:04.196750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.196719 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:27:04.196813 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.196786 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:27:04.196848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.196814 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:27:04.308731 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.308690 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" event={"ID":"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7","Type":"ContainerStarted","Data":"eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce"} Nov 04 03:27:04.308878 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.308751 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" event={"ID":"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7","Type":"ContainerStarted","Data":"18b3eae9f891b1f7fad40142978845ff8f6597aa2d7387d1426d62d268a5f8cf"} Nov 04 03:27:04.431886 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.431836 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podStartSLOduration=1.431822735 podStartE2EDuration="1.431822735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:27:04.430575672 +0000 UTC m=+601.814003721" watchObservedRunningTime="2026-11-04 03:27:04.431822735 +0000 UTC m=+601.815250836" Nov 04 03:27:04.764628 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:04.764582 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:05.766130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:05.766090 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:05.766576 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:05.766151 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:06.764996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:06.764950 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:06.765217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:06.765018 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:07.764352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:07.764316 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:07.764739 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:07.764370 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:08.764794 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:08.764759 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:08.765199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:08.764825 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:09.764955 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:09.764921 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:09.765381 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:09.764976 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:10.764623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:10.764586 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:10.764803 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:10.764674 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:11.765385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:11.765347 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-skdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Nov 04 03:27:11.765790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:11.765416 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Nov 04 03:27:12.858436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:12.858389 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:12.858960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:12.858932 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:12.859905 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:12.859884 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:27:16.353586 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:27:16.353550 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:27:32.190440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:32.190400 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:27:32.492387 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:32.492294 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:27:43.834108 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:43.834075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:27:44.101386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:44.101305 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:27:46.353862 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:27:46.353817 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:27:49.053404 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:49.053366 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:27:49.299639 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:49.299603 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:27:56.116898 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:27:56.116866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:07.962805 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:07.962773 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:16.353776 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:28:16.353695 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:28:46.354278 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:28:46.354229 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:28:53.271104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.271073 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:53.526534 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.526439 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:53.742326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.742298 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf"] Nov 04 03:28:53.745665 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.745649 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766282 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csf42\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766309 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766337 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766415 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766604 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766444 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766604 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766545 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766601 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766641 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.766765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.766701 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.798731 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.798658 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-10f834-openshift-default-dockercfg-4d9rk\"" Nov 04 03:28:53.850577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.850547 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf"] Nov 04 03:28:53.867174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867148 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867181 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867220 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867264 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867502 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867570 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867637 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csf42\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867663 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.867816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.867726 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.868048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.868022 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.868199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.868095 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.868580 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.868249 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.868758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.868713 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.868758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.868747 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.870438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.870417 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.870700 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.870683 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.930688 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.930648 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.930875 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.930850 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csf42\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:53.953045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:53.953009 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:54.058781 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.058700 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:54.079191 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.079166 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29"] Nov 04 03:28:54.082738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.082719 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.141808 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.141727 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-af788a-openshift-default-dockercfg-826jt\"" Nov 04 03:28:54.166113 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.166083 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29"] Nov 04 03:28:54.170648 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170609 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.170770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170679 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.170770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170713 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.170770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170760 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.170920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170799 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.170920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170816 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.170920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170832 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.171083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.170968 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278px\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.171083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.171006 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.271787 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271757 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271809 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271836 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271867 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271900 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271925 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.271949 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.272027 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278px\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.272069 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.272350 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.272389 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.272446 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.272765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.272546 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.273182 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.273153 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.274620 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.274599 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.274736 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.274693 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.275178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.275155 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:54.356959 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.356909 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-278px\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.375642 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.375616 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token\") pod \"e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.399533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.399500 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:54.509808 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.509772 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf"] Nov 04 03:28:54.512703 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:28:54.512667 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2e8ed1_88f7_4235_9ea1_c731665829b2.slice/crio-3e9e378055d5dab215e5456bea2de845b5d56ccf9ffdc7ce9f4789877dc84dc4 WatchSource:0}: Error finding container 3e9e378055d5dab215e5456bea2de845b5d56ccf9ffdc7ce9f4789877dc84dc4: Status 404 returned error can't find the container with id 3e9e378055d5dab215e5456bea2de845b5d56ccf9ffdc7ce9f4789877dc84dc4 Nov 04 03:28:54.514849 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.514814 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:28:54.514957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.514880 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:28:54.514957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.514912 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:28:54.778451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.778426 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29"] Nov 04 03:28:54.780177 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:28:54.780152 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd864b5_ecec_4876_9d89_42083cdfd4cc.slice/crio-9b47ee19a279696d90012304544ae365e344eb73e192badcc8bbc007657143dd WatchSource:0}: Error finding container 9b47ee19a279696d90012304544ae365e344eb73e192badcc8bbc007657143dd: Status 404 returned error can't find the container with id 9b47ee19a279696d90012304544ae365e344eb73e192badcc8bbc007657143dd Nov 04 03:28:54.782377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.782341 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:28:54.782438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.782427 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:28:54.782511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.782495 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:28:54.804758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.804721 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" event={"ID":"4cd864b5-ecec-4876-9d89-42083cdfd4cc","Type":"ContainerStarted","Data":"9b47ee19a279696d90012304544ae365e344eb73e192badcc8bbc007657143dd"} Nov 04 03:28:54.806217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.806194 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" event={"ID":"3d2e8ed1-88f7-4235-9ea1-c731665829b2","Type":"ContainerStarted","Data":"fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc"} Nov 04 03:28:54.806308 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.806229 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" event={"ID":"3d2e8ed1-88f7-4235-9ea1-c731665829b2","Type":"ContainerStarted","Data":"3e9e378055d5dab215e5456bea2de845b5d56ccf9ffdc7ce9f4789877dc84dc4"} Nov 04 03:28:54.929675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.929619 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podStartSLOduration=1.929603287 podStartE2EDuration="1.929603287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:28:54.927641714 +0000 UTC m=+712.311069779" watchObservedRunningTime="2026-11-04 03:28:54.929603287 +0000 UTC m=+712.313031327" Nov 04 03:28:54.975256 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:54.975218 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:55.059043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.058950 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:55.064646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.064618 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:55.223322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.223289 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:55.812987 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.812951 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" event={"ID":"4cd864b5-ecec-4876-9d89-42083cdfd4cc","Type":"ContainerStarted","Data":"0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397"} Nov 04 03:28:55.813389 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.813377 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:55.814179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.814150 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:28:55.935744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:55.935680 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podStartSLOduration=1.9356655360000001 podStartE2EDuration="1.935665536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:28:55.934251781 +0000 UTC m=+713.317679825" watchObservedRunningTime="2026-11-04 03:28:55.935665536 +0000 UTC m=+713.319093570" Nov 04 03:28:56.400259 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:56.400217 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:56.405046 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:56.405022 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:56.476131 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:56.476098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:56.818210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:56.818125 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:56.819024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:56.819006 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:28:57.800902 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:57.800856 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:28:58.107431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:28:58.107399 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:00.519383 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:00.519343 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:00.851801 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:00.851764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:01.110573 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.107626 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:01.365334 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.365246 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:01.562586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.562542 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq"] Nov 04 03:29:01.566198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.566182 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.624535 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.624443 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-10f834\"" Nov 04 03:29:01.640141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.640105 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlxd\" (UniqueName: \"kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd\") pod \"payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.640337 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.640284 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.640797 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.640772 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:01.654816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.654763 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq"] Nov 04 03:29:01.683777 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.683746 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-10f834-dockercfg-9t5c9\"" Nov 04 03:29:01.741555 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.741520 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlxd\" (UniqueName: \"kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd\") pod \"payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.741725 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.741606 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.742229 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.742214 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.804352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.804321 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlxd\" (UniqueName: \"kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd\") pod \"payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.880267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.880189 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:01.893312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:01.893280 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:02.257284 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:02.257048 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq"] Nov 04 03:29:02.260278 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:29:02.260249 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237c6545_0fa9_488c_9db2_88292db0836b.slice/crio-aeb76dbbc0aa58d4046d08789dc9f0bcea2a99f61136e983322c612d238dea8c WatchSource:0}: Error finding container aeb76dbbc0aa58d4046d08789dc9f0bcea2a99f61136e983322c612d238dea8c: Status 404 returned error can't find the container with id aeb76dbbc0aa58d4046d08789dc9f0bcea2a99f61136e983322c612d238dea8c Nov 04 03:29:02.262811 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:02.262792 4959 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 04 03:29:02.847162 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:02.847133 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" event={"ID":"237c6545-0fa9-488c-9db2-88292db0836b","Type":"ContainerStarted","Data":"aeb76dbbc0aa58d4046d08789dc9f0bcea2a99f61136e983322c612d238dea8c"} Nov 04 03:29:04.360371 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:04.360334 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:04.620171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:04.614138 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:05.386690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.386659 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:05.820654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.820624 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst"] Nov 04 03:29:05.825480 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.825435 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.865346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.865312 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" event={"ID":"237c6545-0fa9-488c-9db2-88292db0836b","Type":"ContainerStarted","Data":"9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9"} Nov 04 03:29:05.865514 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.865494 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:05.879440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.879415 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-36ccd3-openshift-default-dockercfg-n5hpn\"" Nov 04 03:29:05.892854 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.892827 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.892982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.892875 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.892982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.892894 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.893093 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.893070 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.893161 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.893145 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.893251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.893186 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.893310 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.893242 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.893310 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.893293 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.893414 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.893324 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shnwf\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.931632 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.931594 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst"] Nov 04 03:29:05.972792 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.972492 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.994824 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.994914 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.994956 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.994987 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.995027 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.995053 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnwf\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.995165 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.995227 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.995256 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.995724 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.996750 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.997125 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:05.999494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:05.999280 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.007512 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.000501 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.019086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.017361 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.019086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.019047 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.062232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.062202 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.062371 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.062325 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnwf\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf\") pod \"e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.066801 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.066751 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" podStartSLOduration=2.10905913 podStartE2EDuration="5.066733038s" totalImagesPullingTime="2.957673908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:06.06600229 +0000 UTC m=+723.449430333" watchObservedRunningTime="2026-11-04 03:29:06.066733038 +0000 UTC m=+723.450161080" Nov 04 03:29:06.139668 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.139625 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:06.266540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.266505 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:06.284840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.284807 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk"] Nov 04 03:29:06.289280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.289264 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.345564 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.345139 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c5bec0\"" Nov 04 03:29:06.349264 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.347640 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk"] Nov 04 03:29:06.383957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.383911 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c5bec0-dockercfg-v7bvn\"" Nov 04 03:29:06.407873 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.407798 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.408241 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.407923 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb7ld\" (UniqueName: \"kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld\") pod \"payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.439690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.439662 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw"] Nov 04 03:29:06.443228 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.443209 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.498924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.498891 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw"] Nov 04 03:29:06.509461 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.509432 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.509613 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.509503 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4x22\" (UniqueName: \"kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22\") pod \"payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.509683 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.509660 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb7ld\" (UniqueName: \"kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld\") pod \"payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.509744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.509724 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.510224 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.510204 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.552796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.552764 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst"] Nov 04 03:29:06.555338 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:29:06.555308 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87f2944_7006_42a2_b99e_34a92cf65610.slice/crio-cb2fca2a7ceb8930d7007243d1fd38ce2dee84f27a637ac04fd68a5c5dd4517c WatchSource:0}: Error finding container cb2fca2a7ceb8930d7007243d1fd38ce2dee84f27a637ac04fd68a5c5dd4517c: Status 404 returned error can't find the container with id cb2fca2a7ceb8930d7007243d1fd38ce2dee84f27a637ac04fd68a5c5dd4517c Nov 04 03:29:06.557615 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.557589 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:06.557682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.557650 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:06.557682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.557678 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:06.571512 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.571456 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb7ld\" (UniqueName: \"kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld\") pod \"payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.603250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.603225 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:06.610590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.610569 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4x22\" (UniqueName: \"kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22\") pod \"payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.610678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.610652 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.611272 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.611243 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.673279 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.673237 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4x22\" (UniqueName: \"kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22\") pod \"payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.755658 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.755629 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:06.871808 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.871770 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" event={"ID":"c87f2944-7006-42a2-b99e-34a92cf65610","Type":"ContainerStarted","Data":"4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83"} Nov 04 03:29:06.871954 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.871834 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" event={"ID":"c87f2944-7006-42a2-b99e-34a92cf65610","Type":"ContainerStarted","Data":"cb2fca2a7ceb8930d7007243d1fd38ce2dee84f27a637ac04fd68a5c5dd4517c"} Nov 04 03:29:06.982326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:06.982298 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk"] Nov 04 03:29:06.984351 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:29:06.984320 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef40689_e3ac_41df_a040_5bc335bce2d2.slice/crio-bdc9799091cf3d47c77181b31fe6ca6c9e108ce3733738b3306ddd5bc2b5145c WatchSource:0}: Error finding container bdc9799091cf3d47c77181b31fe6ca6c9e108ce3733738b3306ddd5bc2b5145c: Status 404 returned error can't find the container with id bdc9799091cf3d47c77181b31fe6ca6c9e108ce3733738b3306ddd5bc2b5145c Nov 04 03:29:07.003714 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.003654 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podStartSLOduration=2.003637567 podStartE2EDuration="2.003637567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:06.995127692 +0000 UTC m=+724.378555733" watchObservedRunningTime="2026-11-04 03:29:07.003637567 +0000 UTC m=+724.387065606" Nov 04 03:29:07.134532 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.131274 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw"] Nov 04 03:29:07.142791 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.141835 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:07.145201 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:29:07.145157 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d18bff_f0af_4f31_843c_a01eacce1dee.slice/crio-15f43e7402a14c16628d34504db09652047da6d8928017a583b7efdbda91d555 WatchSource:0}: Error finding container 15f43e7402a14c16628d34504db09652047da6d8928017a583b7efdbda91d555: Status 404 returned error can't find the container with id 15f43e7402a14c16628d34504db09652047da6d8928017a583b7efdbda91d555 Nov 04 03:29:07.147838 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.147813 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:07.878335 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.878297 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" event={"ID":"9ef40689-e3ac-41df-a040-5bc335bce2d2","Type":"ContainerStarted","Data":"ba772e25444e56fa59165539b727dc4cea60796373ac8e66f0fdb6fc874e798c"} Nov 04 03:29:07.878822 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.878365 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:07.878822 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.878389 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" event={"ID":"9ef40689-e3ac-41df-a040-5bc335bce2d2","Type":"ContainerStarted","Data":"bdc9799091cf3d47c77181b31fe6ca6c9e108ce3733738b3306ddd5bc2b5145c"} Nov 04 03:29:07.879907 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.879884 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" event={"ID":"94d18bff-f0af-4f31-843c-a01eacce1dee","Type":"ContainerStarted","Data":"fbb094d0f59e5dfd3c4d3703163cae462136e0cbbfd7c7ca8c22d928afba59b0"} Nov 04 03:29:07.879989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.879927 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" event={"ID":"94d18bff-f0af-4f31-843c-a01eacce1dee","Type":"ContainerStarted","Data":"15f43e7402a14c16628d34504db09652047da6d8928017a583b7efdbda91d555"} Nov 04 03:29:07.879989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.879969 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:07.880061 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.880005 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:07.880932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.880914 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:07.995800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:07.995751 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" podStartSLOduration=1.995737762 podStartE2EDuration="1.995737762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:07.995353755 +0000 UTC m=+725.378781796" watchObservedRunningTime="2026-11-04 03:29:07.995737762 +0000 UTC m=+725.379165802" Nov 04 03:29:08.238377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:08.238262 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" podStartSLOduration=2.238246678 podStartE2EDuration="2.238246678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:08.116070695 +0000 UTC m=+725.499498735" watchObservedRunningTime="2026-11-04 03:29:08.238246678 +0000 UTC m=+725.621674721" Nov 04 03:29:08.877580 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:08.877537 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:09.189712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:09.189637 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:09.450949 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:09.450867 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:09.714297 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:09.714202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:10.958289 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:10.958245 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:11.213230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:11.213138 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:14.848819 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:14.848779 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:15.513703 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:15.513661 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:29:15.557634 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:15.547886 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:15.842888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:15.842854 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:16.358861 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:29:16.358827 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:29:16.875292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:16.875259 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:29:17.138996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:17.138910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:17.408973 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:17.408869 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:18.904341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:18.904310 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:29:18.904814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:18.904388 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:29:20.213605 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.213574 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:20.462885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.462852 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:20.709719 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.709682 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5"] Nov 04 03:29:20.711805 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.711777 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:20.717456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.717434 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:20.779528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.772913 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5"] Nov 04 03:29:20.829684 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.829632 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-af788a-dockercfg-zrzmk\"" Nov 04 03:29:20.865781 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.865750 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zrg\" (UniqueName: \"kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg\") pod \"payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:20.865970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.865797 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:20.868662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.868637 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj"] Nov 04 03:29:20.872249 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.872235 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:20.875785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.875768 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-af788a\"" Nov 04 03:29:20.931215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.931185 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj"] Nov 04 03:29:20.966820 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.966734 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5q9\" (UniqueName: \"kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9\") pod \"payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:20.966989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.966904 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:20.966989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.966945 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zrg\" (UniqueName: \"kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg\") pod \"payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:20.967104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.967021 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:20.967732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:20.967709 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:21.030894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.030863 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zrg\" (UniqueName: \"kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg\") pod \"payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:21.068325 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.068292 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:21.068460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.068374 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5q9\" (UniqueName: \"kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9\") pod \"payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:21.069027 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.069001 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:21.131138 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.131105 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5q9\" (UniqueName: \"kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9\") pod \"payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:21.205078 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.205032 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:21.330964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.330918 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:21.600515 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.581324 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj"] Nov 04 03:29:21.606561 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:29:21.606521 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a0d6d2_3ac5_44c5_bb60_03d10fc0e089.slice/crio-48ce2b81c7d20d922dd23db487064c3a02cd59bcc2cc315dbc49a67b2976a8e8 WatchSource:0}: Error finding container 48ce2b81c7d20d922dd23db487064c3a02cd59bcc2cc315dbc49a67b2976a8e8: Status 404 returned error can't find the container with id 48ce2b81c7d20d922dd23db487064c3a02cd59bcc2cc315dbc49a67b2976a8e8 Nov 04 03:29:21.730649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.730580 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5"] Nov 04 03:29:21.733365 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:29:21.733341 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6c0eec_e13e_4967_aef0_78c0351f2333.slice/crio-647e98f84d49b9f5cc9403e4f11808fa5570e7cb7235ee3c24eb9f3c79fbbce9 WatchSource:0}: Error finding container 647e98f84d49b9f5cc9403e4f11808fa5570e7cb7235ee3c24eb9f3c79fbbce9: Status 404 returned error can't find the container with id 647e98f84d49b9f5cc9403e4f11808fa5570e7cb7235ee3c24eb9f3c79fbbce9 Nov 04 03:29:21.951014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.950917 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" event={"ID":"7b6c0eec-e13e-4967-aef0-78c0351f2333","Type":"ContainerStarted","Data":"5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e"} Nov 04 03:29:21.951014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.950978 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" event={"ID":"7b6c0eec-e13e-4967-aef0-78c0351f2333","Type":"ContainerStarted","Data":"647e98f84d49b9f5cc9403e4f11808fa5570e7cb7235ee3c24eb9f3c79fbbce9"} Nov 04 03:29:21.951233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.951087 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:21.952586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.952559 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" event={"ID":"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089","Type":"ContainerStarted","Data":"d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9"} Nov 04 03:29:21.952711 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.952606 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" event={"ID":"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089","Type":"ContainerStarted","Data":"48ce2b81c7d20d922dd23db487064c3a02cd59bcc2cc315dbc49a67b2976a8e8"} Nov 04 03:29:21.952711 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:21.952629 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:22.068577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:22.068516 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" podStartSLOduration=2.068501836 podStartE2EDuration="2.068501836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:22.067566472 +0000 UTC m=+739.450994515" watchObservedRunningTime="2026-11-04 03:29:22.068501836 +0000 UTC m=+739.451929875" Nov 04 03:29:22.204862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:22.201365 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" podStartSLOduration=2.201345803 podStartE2EDuration="2.201345803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:29:22.200159589 +0000 UTC m=+739.583587630" watchObservedRunningTime="2026-11-04 03:29:22.201345803 +0000 UTC m=+739.584773845" Nov 04 03:29:22.541431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:22.541352 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:29:25.216185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.216102 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:25.467072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.467032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:25.720910 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.720771 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:25.814845 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.814816 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:29:25.815016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.814876 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 03:29:25.975601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.975520 4959 generic.go:356] "Generic (PLEG): container finished" podID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerID="fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc" exitCode=0 Nov 04 03:29:25.975601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.975585 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" event={"ID":"3d2e8ed1-88f7-4235-9ea1-c731665829b2","Type":"ContainerDied","Data":"fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc"} Nov 04 03:29:25.975982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.975966 4959 scope.go:122] "RemoveContainer" containerID="fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc" Nov 04 03:29:25.977212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.977184 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:25.977313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.977248 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:25.977313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:25.977282 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:27.008629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:27.008594 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" event={"ID":"3d2e8ed1-88f7-4235-9ea1-c731665829b2","Type":"ContainerStarted","Data":"080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8"} Nov 04 03:29:27.059369 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:27.059338 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:29:27.059598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:27.059575 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:29:27.059654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:27.059636 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 03:29:27.755041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:27.755005 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:28.060048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:28.059964 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:29:28.060048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:28.060010 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 03:29:28.400495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:28.396865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:28.655647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:28.655542 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:29.059569 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:29.059461 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:29:29.059569 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:29.059545 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 03:29:30.060033 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:30.059998 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:29:30.060424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:30.060068 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 03:29:30.499558 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:30.499438 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:30.759712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:30.759627 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:31.059786 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:31.059691 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Nov 04 03:29:31.059786 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:31.059762 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Nov 04 03:29:32.114456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:32.114416 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:29:32.115551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:32.115515 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:29:32.116550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:32.116529 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:29:32.254239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:32.254203 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:32.962488 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:32.962441 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:29:32.962652 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:32.962540 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:29:35.660569 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:35.660531 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:39.038972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:39.038939 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:29:40.409494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:40.405601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:40.652019 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:40.651985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:45.698548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:45.698413 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:46.364968 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:29:46.364924 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:29:49.167782 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.167752 4959 generic.go:356] "Generic (PLEG): container finished" podID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerID="0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397" exitCode=0 Nov 04 03:29:49.168194 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.167820 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" event={"ID":"4cd864b5-ecec-4876-9d89-42083cdfd4cc","Type":"ContainerDied","Data":"0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397"} Nov 04 03:29:49.168274 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.168258 4959 scope.go:122] "RemoveContainer" containerID="0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397" Nov 04 03:29:49.169650 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.169625 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:49.169730 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.169647 4959 generic.go:356] "Generic (PLEG): container finished" podID="c87f2944-7006-42a2-b99e-34a92cf65610" containerID="4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83" exitCode=0 Nov 04 03:29:49.169730 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.169694 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" event={"ID":"c87f2944-7006-42a2-b99e-34a92cf65610","Type":"ContainerDied","Data":"4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83"} Nov 04 03:29:49.169798 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.169695 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:49.169798 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.169794 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:49.169974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.169961 4959 scope.go:122] "RemoveContainer" containerID="4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83" Nov 04 03:29:49.171036 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.171012 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:49.171139 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.171088 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:49.171200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:49.171139 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:29:50.177900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:50.177869 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" event={"ID":"c87f2944-7006-42a2-b99e-34a92cf65610","Type":"ContainerStarted","Data":"1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9"} Nov 04 03:29:50.207495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:50.206561 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" event={"ID":"4cd864b5-ecec-4876-9d89-42083cdfd4cc","Type":"ContainerStarted","Data":"a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2"} Nov 04 03:29:50.402599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:50.402512 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:29:50.402924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:50.402900 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:29:50.403086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:50.403061 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 04 03:29:50.750337 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:50.750286 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:51.140591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:51.140556 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:51.141806 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:51.141782 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:51.141876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:51.141832 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 04 03:29:51.405943 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:51.405852 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:29:51.406573 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:51.406537 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 04 03:29:51.436533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:51.436497 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:29:52.140591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:52.140557 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:52.140785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:52.140613 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 04 03:29:52.401011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:52.400912 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:29:52.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:52.401221 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 04 03:29:52.883495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:52.882622 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:53.140817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:53.140722 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:53.140817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:53.140776 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 04 03:29:53.403084 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:53.402980 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:29:53.403376 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:53.403326 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 04 03:29:54.140981 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:54.140937 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:54.141497 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:54.140998 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 04 03:29:54.404552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:54.403878 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Nov 04 03:29:54.404552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:54.403945 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Nov 04 03:29:54.524773 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:54.524728 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:55.120496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.120159 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:55.139931 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.139903 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 04 03:29:55.140154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.140130 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 04 03:29:55.371842 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.371748 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:55.474615 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.474577 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:29:55.475014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.474990 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:29:55.475959 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.475937 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:29:55.651456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:55.651364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:56.144194 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:56.144163 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:56.145260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:56.145237 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:29:56.815769 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:56.815718 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:29:59.480376 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:29:59.480337 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:00.169971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.169934 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4"] Nov 04 03:30:00.173173 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.173148 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.173338 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.173207 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl"] Nov 04 03:30:00.175597 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.175576 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb"] Nov 04 03:30:00.175734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.175707 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.178097 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.178078 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.220865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.220834 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf"] Nov 04 03:30:00.223258 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.223242 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.226127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.226104 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:30:00.271757 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.271716 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4"] Nov 04 03:30:00.281906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.281874 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb"] Nov 04 03:30:00.288703 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.288669 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl"] Nov 04 03:30:00.306570 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.306529 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdjn\" (UniqueName: \"kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn\") pod \"maas-api-key-cleanup-29896050-5bbp4\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.306735 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.306599 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdr5x\" (UniqueName: \"kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x\") pod \"maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.306735 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.306694 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896050-5bbp4\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.306814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.306751 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgcj\" (UniqueName: \"kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj\") pod \"maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.306853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.306824 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.306888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.306856 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.320692 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.320658 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf"] Nov 04 03:30:00.327159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.327121 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:30:00.331577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.331552 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:30:00.408166 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408138 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdjn\" (UniqueName: \"kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn\") pod \"maas-api-key-cleanup-29896050-5bbp4\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.408166 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408174 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.408407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408198 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdr5x\" (UniqueName: \"kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x\") pod \"maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.408407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408231 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896050-5bbp4\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.408407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408321 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgcj\" (UniqueName: \"kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj\") pod \"maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.408407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408353 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.408407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408396 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.408628 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408445 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6dl\" (UniqueName: \"kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl\") pod \"maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.409001 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408983 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.409041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.408992 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29896050-5bbp4\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.409041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.409029 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.469748 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.469674 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdjn\" (UniqueName: \"kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn\") pod \"maas-api-key-cleanup-29896050-5bbp4\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.488785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.488756 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:00.515031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.513668 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.515031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.513859 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6dl\" (UniqueName: \"kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl\") pod \"maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.515360 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.515140 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.523633 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.523587 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgcj\" (UniqueName: \"kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj\") pod \"maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.525674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.525651 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdr5x\" (UniqueName: \"kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x\") pod \"maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.561519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.561463 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:00.578975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.578944 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6dl\" (UniqueName: \"kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl\") pod \"maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.795166 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.795088 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:00.811080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.811055 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:00.835219 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.835184 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:00.882667 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:00.880726 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4"] Nov 04 03:30:00.888000 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:30:00.887733 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c98f5a3_eff7_43d9_9def_374300b73f3e.slice/crio-f0473a5492414e232f6d2ae9c162397190d2a46bb0ff7596e58e5a5dae573c04 WatchSource:0}: Error finding container f0473a5492414e232f6d2ae9c162397190d2a46bb0ff7596e58e5a5dae573c04: Status 404 returned error can't find the container with id f0473a5492414e232f6d2ae9c162397190d2a46bb0ff7596e58e5a5dae573c04 Nov 04 03:30:01.172902 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.172877 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl"] Nov 04 03:30:01.174890 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:30:01.174849 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c6d4db_2922_427d_938d_e2f8f2de4fce.slice/crio-5a54ae50313066d909e6f31a0582edc578d7f226da1867fb8673628f9841b6ae WatchSource:0}: Error finding container 5a54ae50313066d909e6f31a0582edc578d7f226da1867fb8673628f9841b6ae: Status 404 returned error can't find the container with id 5a54ae50313066d909e6f31a0582edc578d7f226da1867fb8673628f9841b6ae Nov 04 03:30:01.221795 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.221768 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb"] Nov 04 03:30:01.226689 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.226660 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf"] Nov 04 03:30:01.227085 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:30:01.227059 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7fb1e5_c20d_4681_b0e2_a08f58ee4339.slice/crio-918ed82fa8c9528da49ad40be298ad9c439e183ae6073dd4c1525f8204276138 WatchSource:0}: Error finding container 918ed82fa8c9528da49ad40be298ad9c439e183ae6073dd4c1525f8204276138: Status 404 returned error can't find the container with id 918ed82fa8c9528da49ad40be298ad9c439e183ae6073dd4c1525f8204276138 Nov 04 03:30:01.231284 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:30:01.231262 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ed2d9d_eb02_446d_aa2b_c9abc577c58e.slice/crio-855b816472a720b258a9085348368238e7c01aa0f9ec79db9b1b03ce55849f2c WatchSource:0}: Error finding container 855b816472a720b258a9085348368238e7c01aa0f9ec79db9b1b03ce55849f2c: Status 404 returned error can't find the container with id 855b816472a720b258a9085348368238e7c01aa0f9ec79db9b1b03ce55849f2c Nov 04 03:30:01.294818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.294790 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerStarted","Data":"918ed82fa8c9528da49ad40be298ad9c439e183ae6073dd4c1525f8204276138"} Nov 04 03:30:01.295931 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.295908 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerStarted","Data":"f0473a5492414e232f6d2ae9c162397190d2a46bb0ff7596e58e5a5dae573c04"} Nov 04 03:30:01.297000 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.296974 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" event={"ID":"e7c6d4db-2922-427d-938d-e2f8f2de4fce","Type":"ContainerStarted","Data":"5a54ae50313066d909e6f31a0582edc578d7f226da1867fb8673628f9841b6ae"} Nov 04 03:30:01.298109 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:01.298091 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" event={"ID":"62ed2d9d-eb02-446d-aa2b-c9abc577c58e","Type":"ContainerStarted","Data":"855b816472a720b258a9085348368238e7c01aa0f9ec79db9b1b03ce55849f2c"} Nov 04 03:30:04.551673 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:04.551582 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:05.150382 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:05.150295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:05.401528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:05.401434 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:05.871416 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:05.871377 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:05.885638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:05.885604 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:30:06.329731 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:06.329666 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerStarted","Data":"5d14ed9d542ddefc963b1ee102dc7a3e6658678b8a548840dd9d8f736d345d49"} Nov 04 03:30:06.510364 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:06.510306 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" podStartSLOduration=1.824932046 podStartE2EDuration="6.510290738s" totalImagesPullingTime="4.685358692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:06.508375377 +0000 UTC m=+783.891803416" watchObservedRunningTime="2026-11-04 03:30:06.510290738 +0000 UTC m=+783.893718778" Nov 04 03:30:07.337529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.337456 4959 generic.go:356] "Generic (PLEG): container finished" podID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerID="b202983a01c3a420ac8747f9d92d467adc9af577fba6ab756421dd9e5048e15e" exitCode=60 Nov 04 03:30:07.337529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.337507 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" event={"ID":"e7c6d4db-2922-427d-938d-e2f8f2de4fce","Type":"ContainerDied","Data":"b202983a01c3a420ac8747f9d92d467adc9af577fba6ab756421dd9e5048e15e"} Nov 04 03:30:07.337958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.337804 4959 scope.go:122] "RemoveContainer" containerID="b202983a01c3a420ac8747f9d92d467adc9af577fba6ab756421dd9e5048e15e" Nov 04 03:30:07.339289 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.339259 4959 generic.go:356] "Generic (PLEG): container finished" podID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerID="82eaeb41bfdc18f3dab44b41040e38638f0fec0c0dc52b5cdf33579d3de6d4cf" exitCode=60 Nov 04 03:30:07.339394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.339351 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" event={"ID":"62ed2d9d-eb02-446d-aa2b-c9abc577c58e","Type":"ContainerDied","Data":"82eaeb41bfdc18f3dab44b41040e38638f0fec0c0dc52b5cdf33579d3de6d4cf"} Nov 04 03:30:07.339545 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.339529 4959 scope.go:122] "RemoveContainer" containerID="82eaeb41bfdc18f3dab44b41040e38638f0fec0c0dc52b5cdf33579d3de6d4cf" Nov 04 03:30:07.340999 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.340976 4959 generic.go:356] "Generic (PLEG): container finished" podID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerID="5d14ed9d542ddefc963b1ee102dc7a3e6658678b8a548840dd9d8f736d345d49" exitCode=60 Nov 04 03:30:07.341106 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.341033 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerDied","Data":"5d14ed9d542ddefc963b1ee102dc7a3e6658678b8a548840dd9d8f736d345d49"} Nov 04 03:30:07.341214 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.341197 4959 scope.go:122] "RemoveContainer" containerID="5d14ed9d542ddefc963b1ee102dc7a3e6658678b8a548840dd9d8f736d345d49" Nov 04 03:30:07.342986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.342967 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerStarted","Data":"a536113260c9e5755171a9295a2ac8780b6181a52f4ed266e97da82161f39c91"} Nov 04 03:30:07.955560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:07.955441 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" podStartSLOduration=2.423788328 podStartE2EDuration="7.955426721s" totalImagesPullingTime="5.531638393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:07.955277311 +0000 UTC m=+785.338705351" watchObservedRunningTime="2026-11-04 03:30:07.955426721 +0000 UTC m=+785.338854760" Nov 04 03:30:08.354026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.353999 4959 generic.go:356] "Generic (PLEG): container finished" podID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerID="5f3970c789a18954c45be9e37794b1f359bf7e2a0042547d7c4881063e51ea9e" exitCode=60 Nov 04 03:30:08.354438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.354060 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" event={"ID":"62ed2d9d-eb02-446d-aa2b-c9abc577c58e","Type":"ContainerDied","Data":"5f3970c789a18954c45be9e37794b1f359bf7e2a0042547d7c4881063e51ea9e"} Nov 04 03:30:08.354438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.354120 4959 scope.go:122] "RemoveContainer" containerID="82eaeb41bfdc18f3dab44b41040e38638f0fec0c0dc52b5cdf33579d3de6d4cf" Nov 04 03:30:08.354438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.354364 4959 scope.go:122] "RemoveContainer" containerID="5f3970c789a18954c45be9e37794b1f359bf7e2a0042547d7c4881063e51ea9e" Nov 04 03:30:08.354676 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:08.354655 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf_odh-ai-gateway-infra(62ed2d9d-eb02-446d-aa2b-c9abc577c58e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" Nov 04 03:30:08.356241 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.356221 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerStarted","Data":"f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de"} Nov 04 03:30:08.357594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.357571 4959 generic.go:356] "Generic (PLEG): container finished" podID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerID="a536113260c9e5755171a9295a2ac8780b6181a52f4ed266e97da82161f39c91" exitCode=60 Nov 04 03:30:08.357678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.357653 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerDied","Data":"a536113260c9e5755171a9295a2ac8780b6181a52f4ed266e97da82161f39c91"} Nov 04 03:30:08.357818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.357806 4959 scope.go:122] "RemoveContainer" containerID="a536113260c9e5755171a9295a2ac8780b6181a52f4ed266e97da82161f39c91" Nov 04 03:30:08.359493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.359455 4959 generic.go:356] "Generic (PLEG): container finished" podID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerID="dfbfa9794df7bd28e284c22c81c6c70c0c9510059e6030c0da49ff377d314245" exitCode=60 Nov 04 03:30:08.359585 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.359509 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" event={"ID":"e7c6d4db-2922-427d-938d-e2f8f2de4fce","Type":"ContainerDied","Data":"dfbfa9794df7bd28e284c22c81c6c70c0c9510059e6030c0da49ff377d314245"} Nov 04 03:30:08.359740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.359726 4959 scope.go:122] "RemoveContainer" containerID="dfbfa9794df7bd28e284c22c81c6c70c0c9510059e6030c0da49ff377d314245" Nov 04 03:30:08.359911 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:08.359893 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl_odh-ai-gateway-infra(e7c6d4db-2922-427d-938d-e2f8f2de4fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" Nov 04 03:30:08.368657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:08.368644 4959 scope.go:122] "RemoveContainer" containerID="b202983a01c3a420ac8747f9d92d467adc9af577fba6ab756421dd9e5048e15e" Nov 04 03:30:09.367397 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.367324 4959 generic.go:356] "Generic (PLEG): container finished" podID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerID="f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de" exitCode=60 Nov 04 03:30:09.367864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.367408 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerDied","Data":"f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de"} Nov 04 03:30:09.367864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.367521 4959 scope.go:122] "RemoveContainer" containerID="5d14ed9d542ddefc963b1ee102dc7a3e6658678b8a548840dd9d8f736d345d49" Nov 04 03:30:09.367864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.367675 4959 scope.go:122] "RemoveContainer" containerID="f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de" Nov 04 03:30:09.368053 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:09.367975 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb_odh-ai-gateway-infra(1c7fb1e5-c20d-4681-b0e2-a08f58ee4339)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" Nov 04 03:30:09.369724 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.369702 4959 generic.go:356] "Generic (PLEG): container finished" podID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerID="2f839a1cc298fe01a9e6fe38cc17668990716e24fbc28be8f1c79cc6f3c9d2e7" exitCode=60 Nov 04 03:30:09.369817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.369750 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerDied","Data":"2f839a1cc298fe01a9e6fe38cc17668990716e24fbc28be8f1c79cc6f3c9d2e7"} Nov 04 03:30:09.370062 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.370041 4959 scope.go:122] "RemoveContainer" containerID="2f839a1cc298fe01a9e6fe38cc17668990716e24fbc28be8f1c79cc6f3c9d2e7" Nov 04 03:30:09.370315 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:09.370293 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896050-5bbp4_odh-ai-gateway-infra(9c98f5a3-eff7-43d9-9def-374300b73f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" Nov 04 03:30:09.371643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.371628 4959 scope.go:122] "RemoveContainer" containerID="dfbfa9794df7bd28e284c22c81c6c70c0c9510059e6030c0da49ff377d314245" Nov 04 03:30:09.371878 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:09.371858 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl_odh-ai-gateway-infra(e7c6d4db-2922-427d-938d-e2f8f2de4fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" Nov 04 03:30:09.373371 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.373352 4959 scope.go:122] "RemoveContainer" containerID="5f3970c789a18954c45be9e37794b1f359bf7e2a0042547d7c4881063e51ea9e" Nov 04 03:30:09.373619 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:09.373598 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf_odh-ai-gateway-infra(62ed2d9d-eb02-446d-aa2b-c9abc577c58e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" Nov 04 03:30:09.379352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:09.379332 4959 scope.go:122] "RemoveContainer" containerID="a536113260c9e5755171a9295a2ac8780b6181a52f4ed266e97da82161f39c91" Nov 04 03:30:10.380292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:10.380264 4959 scope.go:122] "RemoveContainer" containerID="f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de" Nov 04 03:30:10.380742 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:10.380572 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb_odh-ai-gateway-infra(1c7fb1e5-c20d-4681-b0e2-a08f58ee4339)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" Nov 04 03:30:10.381893 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:10.381871 4959 scope.go:122] "RemoveContainer" containerID="2f839a1cc298fe01a9e6fe38cc17668990716e24fbc28be8f1c79cc6f3c9d2e7" Nov 04 03:30:10.382089 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:10.382073 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29896050-5bbp4_odh-ai-gateway-infra(9c98f5a3-eff7-43d9-9def-374300b73f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" Nov 04 03:30:10.768148 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:10.768052 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:11.135834 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:11.135793 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:11.402228 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:11.402145 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:11.648906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:11.648866 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:15.936655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:15.936609 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:16.353855 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:16.353820 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:30:16.986038 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:16.985998 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:17.607430 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:17.607385 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:17.862126 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:17.862039 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:19.792168 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:19.792140 4959 scope.go:122] "RemoveContainer" containerID="dfbfa9794df7bd28e284c22c81c6c70c0c9510059e6030c0da49ff377d314245" Nov 04 03:30:20.434513 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:20.434406 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" event={"ID":"e7c6d4db-2922-427d-938d-e2f8f2de4fce","Type":"ContainerStarted","Data":"1a41469430f3cb658457ecd33845b65fe8d3a40aaac101163f0331047273c586"} Nov 04 03:30:20.601795 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:20.601752 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" podStartSLOduration=15.429076536 podStartE2EDuration="20.601739229s" totalImagesPullingTime="5.172662693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:20.551040253 +0000 UTC m=+797.934468292" watchObservedRunningTime="2026-11-04 03:30:20.601739229 +0000 UTC m=+797.985167291" Nov 04 03:30:20.792879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:20.792788 4959 scope.go:122] "RemoveContainer" containerID="5f3970c789a18954c45be9e37794b1f359bf7e2a0042547d7c4881063e51ea9e" Nov 04 03:30:21.224260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.224212 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:21.441506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.441397 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" event={"ID":"62ed2d9d-eb02-446d-aa2b-c9abc577c58e","Type":"ContainerStarted","Data":"a24ca4942ddcf330a0a30673579ea3825fa87d14347e8042808cb61f0a0c6fbe"} Nov 04 03:30:21.443275 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.443252 4959 generic.go:356] "Generic (PLEG): container finished" podID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerID="1a41469430f3cb658457ecd33845b65fe8d3a40aaac101163f0331047273c586" exitCode=60 Nov 04 03:30:21.443395 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.443306 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" event={"ID":"e7c6d4db-2922-427d-938d-e2f8f2de4fce","Type":"ContainerDied","Data":"1a41469430f3cb658457ecd33845b65fe8d3a40aaac101163f0331047273c586"} Nov 04 03:30:21.443395 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.443343 4959 scope.go:122] "RemoveContainer" containerID="dfbfa9794df7bd28e284c22c81c6c70c0c9510059e6030c0da49ff377d314245" Nov 04 03:30:21.444048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.444025 4959 scope.go:122] "RemoveContainer" containerID="1a41469430f3cb658457ecd33845b65fe8d3a40aaac101163f0331047273c586" Nov 04 03:30:21.444450 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:21.444419 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl_odh-ai-gateway-infra(e7c6d4db-2922-427d-938d-e2f8f2de4fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" Nov 04 03:30:21.562110 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.562078 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl"] Nov 04 03:30:21.612026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.611977 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" podStartSLOduration=16.857268598 podStartE2EDuration="21.611961399s" totalImagesPullingTime="4.754692801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:21.565262494 +0000 UTC m=+798.948690535" watchObservedRunningTime="2026-11-04 03:30:21.611961399 +0000 UTC m=+798.995389440" Nov 04 03:30:21.791913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:21.791830 4959 scope.go:122] "RemoveContainer" containerID="f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de" Nov 04 03:30:22.450409 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.450322 4959 generic.go:356] "Generic (PLEG): container finished" podID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerID="a24ca4942ddcf330a0a30673579ea3825fa87d14347e8042808cb61f0a0c6fbe" exitCode=60 Nov 04 03:30:22.450885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.450405 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" event={"ID":"62ed2d9d-eb02-446d-aa2b-c9abc577c58e","Type":"ContainerDied","Data":"a24ca4942ddcf330a0a30673579ea3825fa87d14347e8042808cb61f0a0c6fbe"} Nov 04 03:30:22.450885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.450482 4959 scope.go:122] "RemoveContainer" containerID="5f3970c789a18954c45be9e37794b1f359bf7e2a0042547d7c4881063e51ea9e" Nov 04 03:30:22.450885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.450809 4959 scope.go:122] "RemoveContainer" containerID="a24ca4942ddcf330a0a30673579ea3825fa87d14347e8042808cb61f0a0c6fbe" Nov 04 03:30:22.451128 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:22.451105 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf_odh-ai-gateway-infra(62ed2d9d-eb02-446d-aa2b-c9abc577c58e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" Nov 04 03:30:22.452358 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.452332 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerStarted","Data":"d3082fd78764a99be6af8cdb486a70727e3582444c0ee32cf12546c44bee59bc"} Nov 04 03:30:22.597231 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.597208 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:22.621672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.621649 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf"] Nov 04 03:30:22.749608 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.749546 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca\") pod \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " Nov 04 03:30:22.749738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.749650 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x\" (UniqueName: \"kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x\") pod \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\" (UID: \"e7c6d4db-2922-427d-938d-e2f8f2de4fce\") " Nov 04 03:30:22.749889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.749867 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca" pod "e7c6d4db-2922-427d-938d-e2f8f2de4fce" (UID: "e7c6d4db-2922-427d-938d-e2f8f2de4fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:22.749988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.749971 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7c6d4db-2922-427d-938d-e2f8f2de4fce-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:22.751843 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.751820 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x" pod "e7c6d4db-2922-427d-938d-e2f8f2de4fce" (UID: "e7c6d4db-2922-427d-938d-e2f8f2de4fce"). InnerVolumeSpecName "kube-api-access-zdr5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:22.798291 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.798266 4959 scope.go:122] "RemoveContainer" containerID="2f839a1cc298fe01a9e6fe38cc17668990716e24fbc28be8f1c79cc6f3c9d2e7" Nov 04 03:30:22.850567 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:22.850527 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdr5x\" (UniqueName: \"kubernetes.io/projected/e7c6d4db-2922-427d-938d-e2f8f2de4fce-kube-api-access-zdr5x\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:23.462345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.462312 4959 generic.go:356] "Generic (PLEG): container finished" podID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerID="d3082fd78764a99be6af8cdb486a70727e3582444c0ee32cf12546c44bee59bc" exitCode=60 Nov 04 03:30:23.462761 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.462380 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerDied","Data":"d3082fd78764a99be6af8cdb486a70727e3582444c0ee32cf12546c44bee59bc"} Nov 04 03:30:23.462761 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.462446 4959 scope.go:122] "RemoveContainer" containerID="f82e1baf84c2edde6b751d96895fdc34f4927312f107562f22d9106331b195de" Nov 04 03:30:23.462865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.462845 4959 scope.go:122] "RemoveContainer" containerID="d3082fd78764a99be6af8cdb486a70727e3582444c0ee32cf12546c44bee59bc" Nov 04 03:30:23.463203 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:23.463176 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb_odh-ai-gateway-infra(1c7fb1e5-c20d-4681-b0e2-a08f58ee4339)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" Nov 04 03:30:23.464533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.464513 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerStarted","Data":"f38152d84536f869bfd20728d565f6efce8eb62aa901d134dc2094f272c5b85f"} Nov 04 03:30:23.465745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.465727 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" Nov 04 03:30:23.465745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.465739 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl" event={"ID":"e7c6d4db-2922-427d-938d-e2f8f2de4fce","Type":"ContainerDied","Data":"5a54ae50313066d909e6f31a0582edc578d7f226da1867fb8673628f9841b6ae"} Nov 04 03:30:23.474212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.474188 4959 scope.go:122] "RemoveContainer" containerID="1a41469430f3cb658457ecd33845b65fe8d3a40aaac101163f0331047273c586" Nov 04 03:30:23.611308 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.611278 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:23.693600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.693577 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl"] Nov 04 03:30:23.744015 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.743945 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-90051a-29896050-xddbl"] Nov 04 03:30:23.758393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.758372 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl\" (UniqueName: \"kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl\") pod \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " Nov 04 03:30:23.758500 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.758440 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca\") pod \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\" (UID: \"62ed2d9d-eb02-446d-aa2b-c9abc577c58e\") " Nov 04 03:30:23.758771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.758751 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca" pod "62ed2d9d-eb02-446d-aa2b-c9abc577c58e" (UID: "62ed2d9d-eb02-446d-aa2b-c9abc577c58e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:23.760594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.760572 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl" pod "62ed2d9d-eb02-446d-aa2b-c9abc577c58e" (UID: "62ed2d9d-eb02-446d-aa2b-c9abc577c58e"). InnerVolumeSpecName "kube-api-access-pc6dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:23.810091 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.810061 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb"] Nov 04 03:30:23.859547 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.859518 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:23.859547 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:23.859547 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6dl\" (UniqueName: \"kubernetes.io/projected/62ed2d9d-eb02-446d-aa2b-c9abc577c58e-kube-api-access-pc6dl\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:24.472018 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.471990 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" Nov 04 03:30:24.472487 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.472021 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf" event={"ID":"62ed2d9d-eb02-446d-aa2b-c9abc577c58e","Type":"ContainerDied","Data":"855b816472a720b258a9085348368238e7c01aa0f9ec79db9b1b03ce55849f2c"} Nov 04 03:30:24.472487 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.472085 4959 scope.go:122] "RemoveContainer" containerID="a24ca4942ddcf330a0a30673579ea3825fa87d14347e8042808cb61f0a0c6fbe" Nov 04 03:30:24.477263 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.477220 4959 generic.go:356] "Generic (PLEG): container finished" podID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerID="f38152d84536f869bfd20728d565f6efce8eb62aa901d134dc2094f272c5b85f" exitCode=60 Nov 04 03:30:24.477401 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.477290 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerDied","Data":"f38152d84536f869bfd20728d565f6efce8eb62aa901d134dc2094f272c5b85f"} Nov 04 03:30:24.477723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.477655 4959 scope.go:122] "RemoveContainer" containerID="f38152d84536f869bfd20728d565f6efce8eb62aa901d134dc2094f272c5b85f" Nov 04 03:30:24.477932 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:24.477914 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29896050-5bbp4_odh-ai-gateway-infra(9c98f5a3-eff7-43d9-9def-374300b73f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" Nov 04 03:30:24.482746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.482729 4959 scope.go:122] "RemoveContainer" containerID="2f839a1cc298fe01a9e6fe38cc17668990716e24fbc28be8f1c79cc6f3c9d2e7" Nov 04 03:30:24.612044 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.612016 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:24.637960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.637929 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf"] Nov 04 03:30:24.646804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.643844 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36ccd3-29896050-r9frf"] Nov 04 03:30:24.668311 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.668272 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca\") pod \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " Nov 04 03:30:24.668444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.668393 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj\" (UniqueName: \"kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj\") pod \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\" (UID: \"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339\") " Nov 04 03:30:24.668662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.668640 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca" pod "1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" (UID: "1c7fb1e5-c20d-4681-b0e2-a08f58ee4339"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:24.670597 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.670572 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj" pod "1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" (UID: "1c7fb1e5-c20d-4681-b0e2-a08f58ee4339"). InnerVolumeSpecName "kube-api-access-9lgcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:24.769553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.769456 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lgcj\" (UniqueName: \"kubernetes.io/projected/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-kube-api-access-9lgcj\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:24.769553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.769508 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:24.797691 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.797656 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" path="/var/lib/kubelet/pods/62ed2d9d-eb02-446d-aa2b-c9abc577c58e/volumes" Nov 04 03:30:24.798151 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.798130 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" path="/var/lib/kubelet/pods/e7c6d4db-2922-427d-938d-e2f8f2de4fce/volumes" Nov 04 03:30:24.979301 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.979251 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:24.984073 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:24.984048 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4"] Nov 04 03:30:25.486586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.486556 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" event={"ID":"1c7fb1e5-c20d-4681-b0e2-a08f58ee4339","Type":"ContainerDied","Data":"918ed82fa8c9528da49ad40be298ad9c439e183ae6073dd4c1525f8204276138"} Nov 04 03:30:25.487044 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.486613 4959 scope.go:122] "RemoveContainer" containerID="d3082fd78764a99be6af8cdb486a70727e3582444c0ee32cf12546c44bee59bc" Nov 04 03:30:25.487044 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.486565 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb" Nov 04 03:30:25.634041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.634018 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:25.653588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.653562 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb"] Nov 04 03:30:25.657813 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.657792 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-10f834-29896050-rzjtb"] Nov 04 03:30:25.779760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.779663 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn\" (UniqueName: \"kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn\") pod \"9c98f5a3-eff7-43d9-9def-374300b73f3e\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " Nov 04 03:30:25.779937 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.779837 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca\") pod \"9c98f5a3-eff7-43d9-9def-374300b73f3e\" (UID: \"9c98f5a3-eff7-43d9-9def-374300b73f3e\") " Nov 04 03:30:25.780184 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.780158 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca" pod "9c98f5a3-eff7-43d9-9def-374300b73f3e" (UID: "9c98f5a3-eff7-43d9-9def-374300b73f3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:30:25.781894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.781875 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn" pod "9c98f5a3-eff7-43d9-9def-374300b73f3e" (UID: "9c98f5a3-eff7-43d9-9def-374300b73f3e"). InnerVolumeSpecName "kube-api-access-8zdjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:30:25.880734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.880708 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c98f5a3-eff7-43d9-9def-374300b73f3e-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:25.880734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:25.880734 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdjn\" (UniqueName: \"kubernetes.io/projected/9c98f5a3-eff7-43d9-9def-374300b73f3e-kube-api-access-8zdjn\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:30:26.036424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.036324 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:26.497738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.497707 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" Nov 04 03:30:26.498187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.497740 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4" event={"ID":"9c98f5a3-eff7-43d9-9def-374300b73f3e","Type":"ContainerDied","Data":"f0473a5492414e232f6d2ae9c162397190d2a46bb0ff7596e58e5a5dae573c04"} Nov 04 03:30:26.498187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.497816 4959 scope.go:122] "RemoveContainer" containerID="f38152d84536f869bfd20728d565f6efce8eb62aa901d134dc2094f272c5b85f" Nov 04 03:30:26.670851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.670821 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4"] Nov 04 03:30:26.721200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.721172 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29896050-5bbp4"] Nov 04 03:30:26.798523 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.798443 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" path="/var/lib/kubelet/pods/1c7fb1e5-c20d-4681-b0e2-a08f58ee4339/volumes" Nov 04 03:30:26.798844 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:26.798830 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" path="/var/lib/kubelet/pods/9c98f5a3-eff7-43d9-9def-374300b73f3e/volumes" Nov 04 03:30:28.116503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116461 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p"] Nov 04 03:30:28.116889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116869 4959 cpu_manager.go:414] "containerMap: removing container" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.116889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116880 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.116889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116888 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116894 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116905 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116911 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116916 4959 cpu_manager.go:414] "containerMap: removing container" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116920 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116929 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116933 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116957 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116962 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116970 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116974 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116979 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:28.116989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.116985 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117042 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117050 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117055 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117060 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117067 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117071 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117077 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117081 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.117341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.117088 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:28.121549 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.121534 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.201519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.201487 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svp4b\" (UniqueName: \"kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.201677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.201541 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.201677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.201622 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.201771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.201730 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.201771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.201761 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.201861 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.201796 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.226914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.226873 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-af788a\"/\"default-dockercfg-22bhm\"" Nov 04 03:30:28.226914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.226901 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-af788a\"/\"openshift-service-ca.crt\"" Nov 04 03:30:28.226914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.226900 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-af788a\"/\"kube-root-ca.crt\"" Nov 04 03:30:28.227228 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.226964 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-af788a\"/\"simulator-w2-af788a-kserve-self-signed-certs\"" Nov 04 03:30:28.229347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.229324 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p"] Nov 04 03:30:28.302686 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.302652 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svp4b\" (UniqueName: \"kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.302686 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.302692 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.302879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.302725 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.302879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.302795 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.302879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.302824 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.302879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.302868 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.303230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.303206 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.303306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.303215 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.303306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.303277 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.305068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.305033 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.305303 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.305287 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.307967 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.307912 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:28.368420 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.365956 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svp4b\" (UniqueName: \"kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b\") pod \"simulator-w2-af788a-kserve-74bf9d47bd-c5s6p\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.432938 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.432908 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:28.812508 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:28.812484 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p"] Nov 04 03:30:28.814448 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:30:28.814424 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0e09e3f_9b31_40cf_93ec_23ca6df4fe2c.slice/crio-e1ccff9619d663d8e0177211b8fcb0a5b9ac0cc9afc73d73f606e0fb919b152d WatchSource:0}: Error finding container e1ccff9619d663d8e0177211b8fcb0a5b9ac0cc9afc73d73f606e0fb919b152d: Status 404 returned error can't find the container with id e1ccff9619d663d8e0177211b8fcb0a5b9ac0cc9afc73d73f606e0fb919b152d Nov 04 03:30:29.515692 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:29.515660 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" event={"ID":"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c","Type":"ContainerStarted","Data":"0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995"} Nov 04 03:30:29.516082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:29.515703 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" event={"ID":"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c","Type":"ContainerStarted","Data":"e1ccff9619d663d8e0177211b8fcb0a5b9ac0cc9afc73d73f606e0fb919b152d"} Nov 04 03:30:29.637247 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:29.637195 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" podStartSLOduration=1.637182663 podStartE2EDuration="1.637182663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:29.634891055 +0000 UTC m=+807.018319097" watchObservedRunningTime="2026-11-04 03:30:29.637182663 +0000 UTC m=+807.020610717" Nov 04 03:30:30.433588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:30.433550 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:30.858628 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:30.858594 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:32.544217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:32.544184 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:30:35.720973 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:35.720935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:37.980233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:37.980174 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:38.434054 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:38.434015 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:38.446981 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:38.446957 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:38.572260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:38.572232 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:30:40.523752 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:40.523719 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:45.464023 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:45.463983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:46.354597 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:30:46.354557 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:30:46.764732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.764658 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65"] Nov 04 03:30:46.765091 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765070 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:46.765091 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765080 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:46.765158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765101 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:46.765158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765107 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:46.765158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765113 4959 cpu_manager.go:414] "containerMap: removing container" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:46.765158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765117 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ed2d9d-eb02-446d-aa2b-c9abc577c58e" containerName="cleanup" Nov 04 03:30:46.765158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765125 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:46.765158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765129 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:46.765590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765207 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c6d4db-2922-427d-938d-e2f8f2de4fce" containerName="cleanup" Nov 04 03:30:46.765590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765214 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c98f5a3-eff7-43d9-9def-374300b73f3e" containerName="cleanup" Nov 04 03:30:46.765590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.765221 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7fb1e5-c20d-4681-b0e2-a08f58ee4339" containerName="cleanup" Nov 04 03:30:46.768121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.768105 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.869782 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.869743 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhsq\" (UniqueName: \"kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.869969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.869809 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.869969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.869834 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.869969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.869868 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.869969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.869889 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.870119 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.870072 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.871449 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.871429 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-90051a\"/\"default-dockercfg-mtpqk\"" Nov 04 03:30:46.871522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.871494 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-90051a\"/\"kube-root-ca.crt\"" Nov 04 03:30:46.871522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.871510 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-90051a\"/\"simulator-w1-90051a-kserve-self-signed-certs\"" Nov 04 03:30:46.871603 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.871575 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-90051a\"/\"openshift-service-ca.crt\"" Nov 04 03:30:46.873751 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.873733 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65"] Nov 04 03:30:46.966796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.966761 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:46.970904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.970885 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.970962 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhsq\" (UniqueName: \"kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.970997 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.971011 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971181 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.971036 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971181 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.971052 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971354 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.971335 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971400 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.971356 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.971464 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.971446 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.973322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.973302 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:46.973454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:46.973437 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:47.038147 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:47.038075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhsq\" (UniqueName: \"kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq\") pod \"simulator-w1-90051a-kserve-f8566f44-bsx65\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:47.080039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:47.080009 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:47.452927 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:47.452904 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65"] Nov 04 03:30:47.454591 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:30:47.454566 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69022d9a_0412_41c2_80f0_25cd144baedf.slice/crio-c028db99b187866ea86eca675d2bd9283772756c3e747db8e37901bfcbc49279 WatchSource:0}: Error finding container c028db99b187866ea86eca675d2bd9283772756c3e747db8e37901bfcbc49279: Status 404 returned error can't find the container with id c028db99b187866ea86eca675d2bd9283772756c3e747db8e37901bfcbc49279 Nov 04 03:30:47.607942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:47.607903 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" event={"ID":"69022d9a-0412-41c2-80f0-25cd144baedf","Type":"ContainerStarted","Data":"b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0"} Nov 04 03:30:47.607942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:47.607948 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" event={"ID":"69022d9a-0412-41c2-80f0-25cd144baedf","Type":"ContainerStarted","Data":"c028db99b187866ea86eca675d2bd9283772756c3e747db8e37901bfcbc49279"} Nov 04 03:30:47.726793 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:47.726681 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" podStartSLOduration=1.726667204 podStartE2EDuration="1.726667204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:30:47.725845318 +0000 UTC m=+825.109273362" watchObservedRunningTime="2026-11-04 03:30:47.726667204 +0000 UTC m=+825.110095261" Nov 04 03:30:48.080547 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:48.080514 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:50.446749 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:50.446715 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:55.323842 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:55.323801 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:56.377495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:56.376547 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:30:57.080354 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:57.080316 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:57.092724 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:57.092698 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:30:57.196737 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:57.196701 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:30:57.673217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:30:57.673188 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:31:00.149022 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:00.148986 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:04.927917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:04.927881 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:09.735896 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:09.735860 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:14.533894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:14.533798 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:16.355010 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:31:16.354971 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:31:17.262190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:17.262156 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:19.353777 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:19.353738 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:23.658507 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:23.655636 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:31:24.327699 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:24.327661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:29.238422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:29.238385 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:33.939648 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:33.939612 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:38.136383 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.136345 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9"] Nov 04 03:31:38.140159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.140143 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.184679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.184649 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.184825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.184708 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.184825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.184787 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.184912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.184840 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbxv\" (UniqueName: \"kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.184912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.184895 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.184984 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.184952 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.239090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.239054 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:38.243688 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.243657 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-af788a\"/\"premium-w2-af788a-kserve-self-signed-certs\"" Nov 04 03:31:38.247983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.247955 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9"] Nov 04 03:31:38.289375 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.289330 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.289623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.289431 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.289623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.289498 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.289623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.289565 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbxv\" (UniqueName: \"kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.289623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.289596 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.289623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.289623 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.290458 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.290402 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.290939 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.290906 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.292984 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.292959 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.294554 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.294532 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.299121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.299093 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.353403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.353366 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbxv\" (UniqueName: \"kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv\") pod \"premium-w2-af788a-kserve-7cb76777df-57gp9\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.452332 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.452261 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:38.738333 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.738259 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:38.852117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.852094 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9"] Nov 04 03:31:38.854326 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:31:38.854301 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9db5325c_c80b_4633_9d97_792b69a56bac.slice/crio-548209c7df6606c3f1ed768971e7742ae08420133ea11e0a24aa8af7cca88f59 WatchSource:0}: Error finding container 548209c7df6606c3f1ed768971e7742ae08420133ea11e0a24aa8af7cca88f59: Status 404 returned error can't find the container with id 548209c7df6606c3f1ed768971e7742ae08420133ea11e0a24aa8af7cca88f59 Nov 04 03:31:38.873268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:38.873241 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" event={"ID":"9db5325c-c80b-4633-9d97-792b69a56bac","Type":"ContainerStarted","Data":"548209c7df6606c3f1ed768971e7742ae08420133ea11e0a24aa8af7cca88f59"} Nov 04 03:31:39.879895 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:39.879857 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" event={"ID":"9db5325c-c80b-4633-9d97-792b69a56bac","Type":"ContainerStarted","Data":"cf535f2367a1a2894148414a13adc1fd64a798289ef15cb5d49e49f57690ea69"} Nov 04 03:31:39.998880 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:39.998829 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" podStartSLOduration=1.998814562 podStartE2EDuration="1.998814562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:31:39.996763981 +0000 UTC m=+877.380192021" watchObservedRunningTime="2026-11-04 03:31:39.998814562 +0000 UTC m=+877.382242601" Nov 04 03:31:40.452873 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:40.452838 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:43.154823 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:43.154778 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:43.529889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:43.529802 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:43.789672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:43.789570 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:45.464554 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:45.464523 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:31:46.355035 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:31:46.355003 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:31:48.426985 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:48.426940 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:48.452818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:48.452786 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:48.465725 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:48.465702 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:48.936679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:48.936648 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:31:50.325410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.325381 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj"] Nov 04 03:31:50.329187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.329168 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.416436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.416398 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtqh5\" (UniqueName: \"kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.416612 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.416558 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.416612 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.416583 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.416612 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.416605 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.416732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.416717 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.416770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.416754 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.432957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.432934 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-90051a\"/\"premium-w1-90051a-kserve-self-signed-certs\"" Nov 04 03:31:50.435127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.435106 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj"] Nov 04 03:31:50.512418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.512370 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:50.517573 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.517551 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.517712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.517585 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.517712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.517606 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.517712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.517632 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.517712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.517655 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.517972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.517755 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtqh5\" (UniqueName: \"kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.518026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.518010 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.518090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.518049 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.518148 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.518087 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.520182 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.520151 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.520395 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.520375 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.580349 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.580286 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtqh5\" (UniqueName: \"kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5\") pod \"premium-w1-90051a-kserve-74588668f-hd8bj\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:50.641279 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:50.641247 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:51.060144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:51.060117 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj"] Nov 04 03:31:51.062162 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:31:51.062128 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aeccee1_0227_4f41_a815_0d813e660f76.slice/crio-501ea114f47f8d366d358e694be9e29a6a184f26bc64488799ad0e23ca9ea704 WatchSource:0}: Error finding container 501ea114f47f8d366d358e694be9e29a6a184f26bc64488799ad0e23ca9ea704: Status 404 returned error can't find the container with id 501ea114f47f8d366d358e694be9e29a6a184f26bc64488799ad0e23ca9ea704 Nov 04 03:31:51.941765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:51.941735 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" event={"ID":"2aeccee1-0227-4f41-a815-0d813e660f76","Type":"ContainerStarted","Data":"9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a"} Nov 04 03:31:51.942175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:51.941773 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" event={"ID":"2aeccee1-0227-4f41-a815-0d813e660f76","Type":"ContainerStarted","Data":"501ea114f47f8d366d358e694be9e29a6a184f26bc64488799ad0e23ca9ea704"} Nov 04 03:31:52.059441 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:52.059381 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" podStartSLOduration=2.059367173 podStartE2EDuration="2.059367173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:31:52.059280414 +0000 UTC m=+889.442708454" watchObservedRunningTime="2026-11-04 03:31:52.059367173 +0000 UTC m=+889.442795212" Nov 04 03:31:52.643503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:52.641338 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:31:53.222906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:53.222874 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:57.511313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:57.511272 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:31:58.149563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:31:58.149521 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:00.642428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:00.642391 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:32:00.655211 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:00.655187 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:32:00.998578 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:00.998463 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:32:02.861353 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:02.861326 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:32:02.864130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:02.864106 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:32:02.867887 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:02.867867 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:32:02.870603 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:02.870583 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:32:03.140423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:03.140350 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:07.873592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:07.873539 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:11.052506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:11.052455 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:32:12.687746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:12.687713 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:16.355578 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:32:16.355540 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:32:17.544544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:17.544513 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:18.083847 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.083818 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp"] Nov 04 03:32:18.087456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.087439 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.140786 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.140759 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-90051a-openshift-default-dockercfg-88sl4\"" Nov 04 03:32:18.193780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.193750 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp"] Nov 04 03:32:18.209302 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209261 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209452 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209359 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209452 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209411 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209498 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209530 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrngh\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209559 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209601 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209740 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.209788 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.209758 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310559 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310602 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310619 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrngh\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310640 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310687 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310750 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310776 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.310853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310849 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.311135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.310893 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.311135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.311035 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.311135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.311071 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.311135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.311105 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.311318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.311295 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.311464 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.311445 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.313318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.313292 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.313672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.313650 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.373802 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.373738 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.374028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.374004 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrngh\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh\") pod \"e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.400710 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.400680 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:18.831764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.831739 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp"] Nov 04 03:32:18.833567 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:32:18.833539 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a6af5b_4d34_4822_819b_1e40e917a36c.slice/crio-c5e63337ff1cc571173f439602a684d48d7acb94a206d1299c867a73ac55dca9 WatchSource:0}: Error finding container c5e63337ff1cc571173f439602a684d48d7acb94a206d1299c867a73ac55dca9: Status 404 returned error can't find the container with id c5e63337ff1cc571173f439602a684d48d7acb94a206d1299c867a73ac55dca9 Nov 04 03:32:18.835744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.835713 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:32:18.835840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.835773 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:32:18.835840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:18.835802 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:32:19.082212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:19.082116 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" event={"ID":"71a6af5b-4d34-4822-819b-1e40e917a36c","Type":"ContainerStarted","Data":"b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861"} Nov 04 03:32:19.082212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:19.082174 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" event={"ID":"71a6af5b-4d34-4822-819b-1e40e917a36c","Type":"ContainerStarted","Data":"c5e63337ff1cc571173f439602a684d48d7acb94a206d1299c867a73ac55dca9"} Nov 04 03:32:19.254269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:19.254217 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podStartSLOduration=1.254201653 podStartE2EDuration="1.254201653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:32:19.203095876 +0000 UTC m=+916.586523917" watchObservedRunningTime="2026-11-04 03:32:19.254201653 +0000 UTC m=+916.637629692" Nov 04 03:32:19.403762 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:19.403656 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:19.405500 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:19.405457 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:32:19.405617 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:19.405540 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:32:20.403946 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:20.403906 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:32:20.404379 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:20.403965 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:32:21.403496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:21.402917 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:32:21.403496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:21.402991 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:32:22.319821 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:22.319786 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:22.402977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:22.402938 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:32:22.403312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:22.403281 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:32:23.403904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:23.403866 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:32:23.404306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:23.403933 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:32:24.407942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:24.407903 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Nov 04 03:32:24.408580 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:24.408544 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Nov 04 03:32:25.644339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:25.644300 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:25.645211 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:25.645171 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:25.646107 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:25.646081 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:32:27.135552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:27.135515 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:31.917541 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:31.917504 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:33.077357 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:33.077327 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:33.406659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:33.406555 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:36.740309 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:36.740277 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:39.844542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:39.844508 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:32:41.781450 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:41.781364 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:45.287268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:45.287229 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:46.357811 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:32:46.357776 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:32:46.366550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:46.366521 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:46.648118 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:46.648038 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:51.455601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:51.455566 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:56.234154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:56.234115 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:57.291341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:57.291295 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:32:57.858753 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:32:57.858710 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:01.133703 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:01.133661 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:05.925119 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:05.925085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:06.878296 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:06.878267 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:33:10.743075 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:10.743040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:15.569840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:15.569806 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:16.354380 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:33:16.354343 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:33:20.343377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:20.343343 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:25.138398 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:25.138353 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:27.094780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:27.094741 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:27.339502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:27.339453 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:27.772141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:27.772099 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:28.041558 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:28.038781 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:28.298926 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:28.298840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:29.946812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:29.946781 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:31.849969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:31.849931 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:32.117557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:32.117436 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:32.485001 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:32.484916 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:32.750893 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:32.750816 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:34.113595 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:34.113561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:35.111174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.111141 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:35.365592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.365510 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:35.588683 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.588648 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q"] Nov 04 03:33:35.602611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.602580 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.655521 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.655432 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q"] Nov 04 03:33:35.705771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.705744 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9dda20-dockercfg-lnw45\"" Nov 04 03:33:35.705771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.705747 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9dda20\"" Nov 04 03:33:35.729307 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.729277 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lkc\" (UniqueName: \"kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc\") pod \"payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.729455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.729401 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.749923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.749899 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml"] Nov 04 03:33:35.753722 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.753704 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:35.819559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.819525 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml"] Nov 04 03:33:35.830182 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.830145 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.830343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.830245 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf27k\" (UniqueName: \"kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k\") pod \"payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:35.830343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.830332 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lkc\" (UniqueName: \"kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc\") pod \"payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.830439 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.830348 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:35.830963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.830942 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.891729 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.891693 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lkc\" (UniqueName: \"kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc\") pod \"payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.916331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.916275 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:35.933221 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.933191 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf27k\" (UniqueName: \"kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k\") pod \"payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:35.933348 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.933298 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:35.934102 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.934081 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:35.994997 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:35.994966 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf27k\" (UniqueName: \"kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k\") pod \"payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:36.067631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.067602 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:36.290638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.290613 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q"] Nov 04 03:33:36.292744 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:33:36.292722 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7a5057_08aa_46d1_b41e_b9247da5524a.slice/crio-81ff667aa8c1009818e7772160418c1d91681eaf430c5bb093507f4d0a5b747a WatchSource:0}: Error finding container 81ff667aa8c1009818e7772160418c1d91681eaf430c5bb093507f4d0a5b747a: Status 404 returned error can't find the container with id 81ff667aa8c1009818e7772160418c1d91681eaf430c5bb093507f4d0a5b747a Nov 04 03:33:36.449153 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.449132 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml"] Nov 04 03:33:36.450935 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:33:36.450910 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaf687b_9810_4e14_8622_57cd47096ab1.slice/crio-b301ab8622e5ce7226cbbfd10ce76a90fd1ecfdb3b09ce042d79cbf92587dd8b WatchSource:0}: Error finding container b301ab8622e5ce7226cbbfd10ce76a90fd1ecfdb3b09ce042d79cbf92587dd8b: Status 404 returned error can't find the container with id b301ab8622e5ce7226cbbfd10ce76a90fd1ecfdb3b09ce042d79cbf92587dd8b Nov 04 03:33:36.500140 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.500107 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" event={"ID":"8aaf687b-9810-4e14-8622-57cd47096ab1","Type":"ContainerStarted","Data":"b301ab8622e5ce7226cbbfd10ce76a90fd1ecfdb3b09ce042d79cbf92587dd8b"} Nov 04 03:33:36.501562 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.501536 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" event={"ID":"3e7a5057-08aa-46d1-b41e-b9247da5524a","Type":"ContainerStarted","Data":"5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8"} Nov 04 03:33:36.501654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.501574 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" event={"ID":"3e7a5057-08aa-46d1-b41e-b9247da5524a","Type":"ContainerStarted","Data":"81ff667aa8c1009818e7772160418c1d91681eaf430c5bb093507f4d0a5b747a"} Nov 04 03:33:36.501694 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.501671 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:36.619849 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:36.619799 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" podStartSLOduration=1.619785917 podStartE2EDuration="1.619785917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:36.618441492 +0000 UTC m=+994.001869530" watchObservedRunningTime="2026-11-04 03:33:36.619785917 +0000 UTC m=+994.003213957" Nov 04 03:33:37.508143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:37.508106 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" event={"ID":"8aaf687b-9810-4e14-8622-57cd47096ab1","Type":"ContainerStarted","Data":"5970cd707b4dc34385c075e5d60025f2c72d6e09b413833e70ea5a33c36f8b69"} Nov 04 03:33:37.508568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:37.508256 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:37.627021 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:37.626971 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" podStartSLOduration=2.626957414 podStartE2EDuration="2.626957414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:37.625212138 +0000 UTC m=+995.008640176" watchObservedRunningTime="2026-11-04 03:33:37.626957414 +0000 UTC m=+995.010385449" Nov 04 03:33:38.553894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:38.553862 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:38.808490 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:38.808391 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:39.943118 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:39.943078 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:40.203546 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.199238 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:40.322566 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.322535 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m"] Nov 04 03:33:40.326256 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.326242 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.381642 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.381603 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-141891\"" Nov 04 03:33:40.384248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.384222 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m"] Nov 04 03:33:40.384437 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.384408 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq728\" (UniqueName: \"kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728\") pod \"payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.384638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.384616 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.436631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.436263 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-141891-dockercfg-t4w85\"" Nov 04 03:33:40.472563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.472485 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j"] Nov 04 03:33:40.476267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.476248 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.485432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.485410 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq728\" (UniqueName: \"kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728\") pod \"payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.485543 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.485452 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.486141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.486116 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.536331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.536300 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j"] Nov 04 03:33:40.548312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.548282 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq728\" (UniqueName: \"kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728\") pod \"payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.586907 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.586882 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.587039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.586910 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrsw6\" (UniqueName: \"kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6\") pod \"payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.640361 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.640332 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:40.688251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.688218 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.688251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.688257 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrsw6\" (UniqueName: \"kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6\") pod \"payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.689001 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.688983 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.749951 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.749880 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrsw6\" (UniqueName: \"kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6\") pod \"payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:40.789316 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:40.789284 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:41.014815 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.014785 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m"] Nov 04 03:33:41.016727 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:33:41.016697 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda394455d_d89b_4513_b82b_22db02ea07ea.slice/crio-42240eb6ecc5ba9493e2276d4a8eb35ec4d7bd4bee7835ae7f8e5935cd063dc6 WatchSource:0}: Error finding container 42240eb6ecc5ba9493e2276d4a8eb35ec4d7bd4bee7835ae7f8e5935cd063dc6: Status 404 returned error can't find the container with id 42240eb6ecc5ba9493e2276d4a8eb35ec4d7bd4bee7835ae7f8e5935cd063dc6 Nov 04 03:33:41.164125 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.164096 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j"] Nov 04 03:33:41.165819 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:33:41.165794 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb658291f_ee09_429e_8b9f_5ddf99384ed3.slice/crio-2d9ddf7573516f3b096dc663a9c86d459ea6b72fd283f935aa030598cc344fef WatchSource:0}: Error finding container 2d9ddf7573516f3b096dc663a9c86d459ea6b72fd283f935aa030598cc344fef: Status 404 returned error can't find the container with id 2d9ddf7573516f3b096dc663a9c86d459ea6b72fd283f935aa030598cc344fef Nov 04 03:33:41.532447 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.532405 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" event={"ID":"a394455d-d89b-4513-b82b-22db02ea07ea","Type":"ContainerStarted","Data":"fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38"} Nov 04 03:33:41.532647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.532498 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:41.532647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.532522 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" event={"ID":"a394455d-d89b-4513-b82b-22db02ea07ea","Type":"ContainerStarted","Data":"42240eb6ecc5ba9493e2276d4a8eb35ec4d7bd4bee7835ae7f8e5935cd063dc6"} Nov 04 03:33:41.533970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.533943 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" event={"ID":"b658291f-ee09-429e-8b9f-5ddf99384ed3","Type":"ContainerStarted","Data":"7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60"} Nov 04 03:33:41.534090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.533979 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" event={"ID":"b658291f-ee09-429e-8b9f-5ddf99384ed3","Type":"ContainerStarted","Data":"2d9ddf7573516f3b096dc663a9c86d459ea6b72fd283f935aa030598cc344fef"} Nov 04 03:33:41.534090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.534053 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:41.650360 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.650308 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" podStartSLOduration=1.650289337 podStartE2EDuration="1.650289337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:41.648829673 +0000 UTC m=+999.032257715" watchObservedRunningTime="2026-11-04 03:33:41.650289337 +0000 UTC m=+999.033717375" Nov 04 03:33:41.776116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:41.775822 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" podStartSLOduration=1.775771964 podStartE2EDuration="1.775771964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:33:41.772934458 +0000 UTC m=+999.156362495" watchObservedRunningTime="2026-11-04 03:33:41.775771964 +0000 UTC m=+999.159200037" Nov 04 03:33:42.392679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:42.392636 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:44.765076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:44.765042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:46.323738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:46.323696 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:46.354243 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:33:46.354211 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:33:47.408648 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:47.408608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:47.511892 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:47.511860 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:33:48.517186 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:48.517158 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:33:49.624451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:49.624416 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:50.360091 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:50.360050 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:33:52.542734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:52.542708 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:33:52.543123 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:52.542824 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:33:54.499337 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:54.499297 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:58.211022 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:58.210987 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:58.457208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:58.457175 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:33:59.248618 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:33:59.248580 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:02.545817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:02.545778 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:03.966321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:03.966290 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:04.214032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:04.213998 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:08.888896 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:08.888863 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:09.133998 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:09.133966 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:09.394092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:09.394045 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:13.462908 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:13.462825 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:34:13.841301 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:13.841270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:15.495243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.495212 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-x2dlh-6xbjc"] Nov 04 03:34:15.500548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.500532 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.556936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.556903 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x2dlh-6xbjc"] Nov 04 03:34:15.589893 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.589862 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.590062 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.589913 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.590110 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.590088 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.590200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.590183 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8sf\" (UniqueName: \"kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.690979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.690947 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.691178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.691029 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8sf\" (UniqueName: \"kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.691178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.691105 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.691178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.691148 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.691368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.691322 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.691844 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.691821 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.692025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.692000 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.753009 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.752949 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8sf\" (UniqueName: \"kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf\") pod \"periodic-gathering-x2dlh-6xbjc\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:15.813614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:15.813587 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:34:16.187071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:16.187048 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x2dlh-6xbjc"] Nov 04 03:34:16.188481 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:34:16.188444 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bccf5c7_ef99_4a3c_b5a4_40a6fae6eb8e.slice/crio-73a470080e58e3726170956de18b3273419641f0d04de012a098120456f5c234 WatchSource:0}: Error finding container 73a470080e58e3726170956de18b3273419641f0d04de012a098120456f5c234: Status 404 returned error can't find the container with id 73a470080e58e3726170956de18b3273419641f0d04de012a098120456f5c234 Nov 04 03:34:16.355091 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:34:16.355050 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:34:16.767331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:16.767301 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" event={"ID":"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e","Type":"ContainerStarted","Data":"e75ec948b2ff9277ac2d66634308ce781ae55b7b3b7e7d6d4fff843b4096110a"} Nov 04 03:34:16.767754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:16.767348 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" event={"ID":"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e","Type":"ContainerStarted","Data":"73a470080e58e3726170956de18b3273419641f0d04de012a098120456f5c234"} Nov 04 03:34:16.888066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:16.888000 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" podStartSLOduration=1.887983115 podStartE2EDuration="1.887983115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:16.885463172 +0000 UTC m=+1034.268891214" watchObservedRunningTime="2026-11-04 03:34:16.887983115 +0000 UTC m=+1034.271411157" Nov 04 03:34:18.825763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:18.825726 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:23.346883 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:23.346845 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:23.729889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:23.729800 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:28.661609 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:28.661576 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:31.680269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:31.680231 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:34:33.573940 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:33.573901 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:38.452056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:38.452012 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:42.017082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:42.017044 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:43.248675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:43.248449 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:46.362565 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:34:46.362523 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:34:48.304364 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:48.304334 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:53.035371 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:53.035332 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:53.623201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:53.623163 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:54.588629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.588594 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:54.642856 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.642826 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q"] Nov 04 03:34:54.646650 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.646635 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.700595 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.700563 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-13190c-openshift-default-dockercfg-5w5wt\"" Nov 04 03:34:54.758152 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.758117 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q"] Nov 04 03:34:54.836717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.836684 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.836717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.836723 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.836970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.836757 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.836970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.836895 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.836970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.836947 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.837078 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.837013 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.837078 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.837051 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.837182 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.837085 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.837182 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.837164 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwtp\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.881335 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.881259 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:54.938879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.938837 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.938916 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.938979 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939029 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939075 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939114 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwtp\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939205 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939272 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939344 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.939795 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.939770 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.942979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.941101 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.942979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.941389 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.942979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.942298 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.944603 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.944553 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.945853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.945825 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.946981 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.946936 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:54.985851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.985827 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp"] Nov 04 03:34:54.989769 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:54.989751 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.004099 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.004076 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:55.004217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.004199 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwtp\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp\") pod \"e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:55.047996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.047945 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d1e2ab-openshift-default-dockercfg-dgtx6\"" Nov 04 03:34:55.055387 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.055360 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp"] Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152263 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152315 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152382 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152559 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68z47\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152679 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152778 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152806 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152834 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.154563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.152875 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.190009 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.189962 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:55.254388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254358 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254390 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254504 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254582 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68z47\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254705 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254680 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254717 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254798 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254744 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254798 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254768 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254802 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.254904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.254804 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.255121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.255075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.255121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.255101 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.255332 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.255312 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.255590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.255570 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.257046 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.257018 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.257434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.257414 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.260641 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.260618 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:55.327678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.322336 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68z47\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.327678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.322824 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token\") pod \"e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.394587 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.394557 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:34:55.441424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.441340 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:55.604043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.604009 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:55.712655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.712629 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q"] Nov 04 03:34:55.715535 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:34:55.715506 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655034e2_d0d9_475d_b1b5_e5a412eb9bc5.slice/crio-f6688af9dc6edd3326a5a8c9a4c2b11dc5543bc8baf6e4ac5649cb588a4be8e1 WatchSource:0}: Error finding container f6688af9dc6edd3326a5a8c9a4c2b11dc5543bc8baf6e4ac5649cb588a4be8e1: Status 404 returned error can't find the container with id f6688af9dc6edd3326a5a8c9a4c2b11dc5543bc8baf6e4ac5649cb588a4be8e1 Nov 04 03:34:55.718049 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.718019 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:55.718123 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.718081 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:55.718123 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.718108 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:55.995938 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:55.995911 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp"] Nov 04 03:34:55.998278 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:34:55.998250 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b5a408_6a57_436d_a4c7_10faa9e27fda.slice/crio-6d5339682526da08dbe20536fdd6e604e3b07cf5567e0248399d0a3befa5b4d4 WatchSource:0}: Error finding container 6d5339682526da08dbe20536fdd6e604e3b07cf5567e0248399d0a3befa5b4d4: Status 404 returned error can't find the container with id 6d5339682526da08dbe20536fdd6e604e3b07cf5567e0248399d0a3befa5b4d4 Nov 04 03:34:56.002809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.002770 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:56.002909 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.002815 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" event={"ID":"655034e2-d0d9-475d-b1b5-e5a412eb9bc5","Type":"ContainerStarted","Data":"2611813c2acbae393b0b64416cf8e807dc41775ee540d47a63adb5e7b973d584"} Nov 04 03:34:56.002909 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.002856 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:56.002909 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.002876 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" event={"ID":"655034e2-d0d9-475d-b1b5-e5a412eb9bc5","Type":"ContainerStarted","Data":"f6688af9dc6edd3326a5a8c9a4c2b11dc5543bc8baf6e4ac5649cb588a4be8e1"} Nov 04 03:34:56.002909 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.002898 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:56.125689 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.125629 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podStartSLOduration=2.125609889 podStartE2EDuration="2.125609889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:56.124428391 +0000 UTC m=+1073.507856432" watchObservedRunningTime="2026-11-04 03:34:56.125609889 +0000 UTC m=+1073.509037931" Nov 04 03:34:56.273287 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.273199 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:56.281323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.281296 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:56.610644 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.610605 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g"] Nov 04 03:34:56.614379 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.614354 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.671745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.671723 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c5bec0-openshift-default-dockercfg-fkbcb\"" Nov 04 03:34:56.681655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.681625 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g"] Nov 04 03:34:56.701149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701116 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701161 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701360 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701187 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701360 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701279 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701402 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701444 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701560 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701617 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tps6n\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.701744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.701712 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802487 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802727 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802547 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802727 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802566 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802727 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802589 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802908 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802725 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802908 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802809 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.802908 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802850 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.803077 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802932 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.803077 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802958 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.803077 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.802975 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tps6n\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.803243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.803123 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.803243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.803209 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.803362 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.803338 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.804026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.804002 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.805862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.805830 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.805978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.805844 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.864030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.863972 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.864138 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.864121 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tps6n\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:56.932795 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:56.932556 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:57.047318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.047283 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" event={"ID":"50b5a408-6a57-436d-a4c7-10faa9e27fda","Type":"ContainerStarted","Data":"3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531"} Nov 04 03:34:57.047575 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.047552 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" event={"ID":"50b5a408-6a57-436d-a4c7-10faa9e27fda","Type":"ContainerStarted","Data":"6d5339682526da08dbe20536fdd6e604e3b07cf5567e0248399d0a3befa5b4d4"} Nov 04 03:34:57.047704 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.047691 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:57.048611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.048586 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:34:57.168738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.168626 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podStartSLOduration=3.168606906 podStartE2EDuration="3.168606906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:57.168223792 +0000 UTC m=+1074.551651835" watchObservedRunningTime="2026-11-04 03:34:57.168606906 +0000 UTC m=+1074.552034938" Nov 04 03:34:57.369802 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.369771 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g"] Nov 04 03:34:57.371696 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:34:57.371657 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5c4bab_9e74_4bbe_861d_edb39737dbae.slice/crio-9a5d494ea1400ea10cbac71f6a82aad48cfdeb144ac0390151d3ccc8cffd4b0f WatchSource:0}: Error finding container 9a5d494ea1400ea10cbac71f6a82aad48cfdeb144ac0390151d3ccc8cffd4b0f: Status 404 returned error can't find the container with id 9a5d494ea1400ea10cbac71f6a82aad48cfdeb144ac0390151d3ccc8cffd4b0f Nov 04 03:34:57.374101 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.374071 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:57.374177 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.374136 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:57.374177 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.374164 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:34:57.604646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.604613 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:57.610179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:57.610152 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:58.054165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.054119 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" event={"ID":"3f5c4bab-9e74-4bbe-861d-edb39737dbae","Type":"ContainerStarted","Data":"ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe"} Nov 04 03:34:58.054667 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.054186 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" event={"ID":"3f5c4bab-9e74-4bbe-861d-edb39737dbae","Type":"ContainerStarted","Data":"9a5d494ea1400ea10cbac71f6a82aad48cfdeb144ac0390151d3ccc8cffd4b0f"} Nov 04 03:34:58.054730 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.054712 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:58.055713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.055692 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:34:58.176254 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.176198 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podStartSLOduration=2.176178878 podStartE2EDuration="2.176178878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:34:58.174986071 +0000 UTC m=+1075.558414111" watchObservedRunningTime="2026-11-04 03:34:58.176178878 +0000 UTC m=+1075.559606922" Nov 04 03:34:58.562006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.561976 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:58.834455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.834425 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:58.933386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.933349 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:34:58.935284 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.935248 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:34:58.935418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:58.935300 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 04 03:34:59.108020 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:59.107767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:59.363661 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:59.363578 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:34:59.933956 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:59.933919 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:34:59.934123 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:59.933994 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 04 03:34:59.966290 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:34:59.966257 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:00.319222 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:00.317436 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:00.640810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:00.632981 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:00.932071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:00.931980 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:00.933859 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:00.933828 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:35:00.933975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:00.933881 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 04 03:35:01.214066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:01.213974 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:01.935299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:01.933538 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:35:01.935299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:01.933666 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 04 03:35:02.364460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:02.364427 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:02.620143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:02.620032 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:02.934280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:02.934204 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:35:02.934280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:02.934253 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 04 03:35:03.933841 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:03.933804 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 04 03:35:03.934230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:03.933872 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 04 03:35:04.442586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:04.442554 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:04.937724 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:04.937699 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:35:04.938191 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:04.938172 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:35:04.938867 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:04.938845 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:35:07.203233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.203186 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh"] Nov 04 03:35:07.203753 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.203433 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" containerID="cri-o://eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce" gracePeriod=30 Nov 04 03:35:07.247345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.247312 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:07.504780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.504702 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:07.553525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.553484 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5"] Nov 04 03:35:07.558313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.558295 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.606422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.606397 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.606566 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.606442 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzr8v\" (UniqueName: \"kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v\") pod \"payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.612341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.612310 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d1e2ab-dockercfg-xpm8j\"" Nov 04 03:35:07.612455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.612341 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d1e2ab\"" Nov 04 03:35:07.616300 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.616276 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5"] Nov 04 03:35:07.708154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.707543 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.708154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.707617 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzr8v\" (UniqueName: \"kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v\") pod \"payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.709287 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.709142 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.714539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.710659 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94"] Nov 04 03:35:07.718581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.718554 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:07.762044 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.761969 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94"] Nov 04 03:35:07.770382 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.770355 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzr8v\" (UniqueName: \"kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v\") pod \"payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.809151 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.809118 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:07.809304 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.809289 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkzz\" (UniqueName: \"kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz\") pod \"payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:07.872154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.872124 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:07.910503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.910453 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkzz\" (UniqueName: \"kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz\") pod \"payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:07.910775 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.910751 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:07.911426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.911406 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:07.972714 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:07.972680 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkzz\" (UniqueName: \"kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz\") pod \"payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:08.034259 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:08.034173 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:08.247709 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:08.247682 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5"] Nov 04 03:35:08.250265 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:35:08.250236 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode820bc4a_fecd_4967_aece_47530a023c93.slice/crio-e0df4e8c41c3753aa1ef49df317349105210537df87807c2dcba28165f67cef0 WatchSource:0}: Error finding container e0df4e8c41c3753aa1ef49df317349105210537df87807c2dcba28165f67cef0: Status 404 returned error can't find the container with id e0df4e8c41c3753aa1ef49df317349105210537df87807c2dcba28165f67cef0 Nov 04 03:35:08.410577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:08.410555 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94"] Nov 04 03:35:08.411776 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:35:08.411750 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de1183b_941b_4d68_994d_41260b0df536.slice/crio-21c84ec1b43065ba62e88b89168017a487ffda15c6bfc0af84df0d3cb2ee4024 WatchSource:0}: Error finding container 21c84ec1b43065ba62e88b89168017a487ffda15c6bfc0af84df0d3cb2ee4024: Status 404 returned error can't find the container with id 21c84ec1b43065ba62e88b89168017a487ffda15c6bfc0af84df0d3cb2ee4024 Nov 04 03:35:09.145766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.145722 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" event={"ID":"8de1183b-941b-4d68-994d-41260b0df536","Type":"ContainerStarted","Data":"40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389"} Nov 04 03:35:09.145948 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.145790 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" event={"ID":"8de1183b-941b-4d68-994d-41260b0df536","Type":"ContainerStarted","Data":"21c84ec1b43065ba62e88b89168017a487ffda15c6bfc0af84df0d3cb2ee4024"} Nov 04 03:35:09.145948 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.145816 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:09.147420 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.147391 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" event={"ID":"e820bc4a-fecd-4967-aece-47530a023c93","Type":"ContainerStarted","Data":"6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549"} Nov 04 03:35:09.147556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.147440 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:09.147556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.147463 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" event={"ID":"e820bc4a-fecd-4967-aece-47530a023c93","Type":"ContainerStarted","Data":"e0df4e8c41c3753aa1ef49df317349105210537df87807c2dcba28165f67cef0"} Nov 04 03:35:09.238305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.238263 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:09.266819 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.266764 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" podStartSLOduration=2.266750095 podStartE2EDuration="2.266750095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:09.263873876 +0000 UTC m=+1086.647301916" watchObservedRunningTime="2026-11-04 03:35:09.266750095 +0000 UTC m=+1086.650178134" Nov 04 03:35:09.408541 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.406108 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" podStartSLOduration=2.406087202 podStartE2EDuration="2.406087202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:09.402337689 +0000 UTC m=+1086.785765730" watchObservedRunningTime="2026-11-04 03:35:09.406087202 +0000 UTC m=+1086.789515242" Nov 04 03:35:09.969953 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:09.969897 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:10.225715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:10.225623 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:12.473253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.473198 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:35:12.559533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559507 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.559657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559557 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.559657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559589 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.559657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559609 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.559657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559642 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.559869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559698 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.559869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559770 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:12.559869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559826 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.560008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559868 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.560008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.559942 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:12.560113 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560050 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data\") pod \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\" (UID: \"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7\") " Nov 04 03:35:12.560113 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560086 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:12.560200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560138 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:35:12.560448 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560367 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:12.560448 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560504 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.560448 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560526 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.560770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560587 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.560770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560716 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.560770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.560747 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.562233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.562208 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:35:12.562366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.562343 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:35:12.562513 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.562493 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:12.562569 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.562543 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm" pod "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" (UID: "2bb05ae7-e6f0-4784-883b-ea2d70d40dd7"). InnerVolumeSpecName "kube-api-access-nmkdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:35:12.661513 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.661480 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.661513 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.661514 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.661706 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.661529 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmkdm\" (UniqueName: \"kubernetes.io/projected/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-kube-api-access-nmkdm\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:12.661706 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:12.661543 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:13.170407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.170374 4959 generic.go:356] "Generic (PLEG): container finished" podID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerID="eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce" exitCode=0 Nov 04 03:35:13.170572 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.170433 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" Nov 04 03:35:13.170572 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.170451 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" event={"ID":"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7","Type":"ContainerDied","Data":"eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce"} Nov 04 03:35:13.170572 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.170532 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh" event={"ID":"2bb05ae7-e6f0-4784-883b-ea2d70d40dd7","Type":"ContainerDied","Data":"18b3eae9f891b1f7fad40142978845ff8f6597aa2d7387d1426d62d268a5f8cf"} Nov 04 03:35:13.170572 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.170556 4959 scope.go:122] "RemoveContainer" containerID="eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce" Nov 04 03:35:13.182621 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.182607 4959 scope.go:122] "RemoveContainer" containerID="eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce" Nov 04 03:35:13.182940 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:35:13.182914 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce\": container with ID starting with eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce not found: ID does not exist" containerID="eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce" Nov 04 03:35:13.183043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.182950 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce"} err="failed to get container status \"eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce\": rpc error: code = NotFound desc = could not find container \"eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce\": container with ID starting with eb56be1fa1c37062078d7dd59bd33cf91da80a8dd568f07dfff8763f4e7720ce not found: ID does not exist" Nov 04 03:35:13.356014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.355985 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh"] Nov 04 03:35:13.406346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:13.406317 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-skdkh"] Nov 04 03:35:14.184590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:14.184554 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:14.736155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:14.736122 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:14.799769 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:14.799738 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" path="/var/lib/kubelet/pods/2bb05ae7-e6f0-4784-883b-ea2d70d40dd7/volumes" Nov 04 03:35:15.008432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:15.008355 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:16.354764 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:35:16.354732 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:35:19.295203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.295168 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:19.548052 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.547963 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:19.793324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.793291 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:19.953452 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.953421 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr"] Nov 04 03:35:19.954006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.953992 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" Nov 04 03:35:19.954054 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.954008 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" Nov 04 03:35:19.954104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.954094 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb05ae7-e6f0-4784-883b-ea2d70d40dd7" containerName="istio-proxy" Nov 04 03:35:19.957533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:19.957518 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.053785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.042313 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.053785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.042372 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmkzx\" (UniqueName: \"kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx\") pod \"payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.104706 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.104670 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:20.106872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.106843 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-491fe7-dockercfg-whcms\"" Nov 04 03:35:20.107052 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.106977 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-491fe7\"" Nov 04 03:35:20.110877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.110845 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr"] Nov 04 03:35:20.143092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.143059 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.143271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.143101 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmkzx\" (UniqueName: \"kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx\") pod \"payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.143865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.143845 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.156196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.156172 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:35:20.156329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.156232 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:35:20.158203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.158179 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd"] Nov 04 03:35:20.161995 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.161978 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.177651 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.177626 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd"] Nov 04 03:35:20.205778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.205709 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmkzx\" (UniqueName: \"kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx\") pod \"payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.244094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.244057 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.244296 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.244110 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpb52\" (UniqueName: \"kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52\") pod \"payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.271041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.271008 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:20.345443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.345405 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.345913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.345495 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpb52\" (UniqueName: \"kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52\") pod \"payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.346881 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.346849 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.409841 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.409810 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpb52\" (UniqueName: \"kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52\") pod \"payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.476076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.476005 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:20.657032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.657004 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr"] Nov 04 03:35:20.658676 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:35:20.658649 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568ca1fb_edf3_43af_aa30_08f3515e9bf0.slice/crio-36d21be1ba428cc0258888dae3506942c3f2f4e22802f392a7e1cbd197127ac0 WatchSource:0}: Error finding container 36d21be1ba428cc0258888dae3506942c3f2f4e22802f392a7e1cbd197127ac0: Status 404 returned error can't find the container with id 36d21be1ba428cc0258888dae3506942c3f2f4e22802f392a7e1cbd197127ac0 Nov 04 03:35:20.856682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:20.856654 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd"] Nov 04 03:35:20.857944 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:35:20.857914 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c56253_aa80_4b21_abc7_6832d690aac8.slice/crio-c8160258391651004cdafb99ded42c8b84b92564955ae348a81308b79f6b284a WatchSource:0}: Error finding container c8160258391651004cdafb99ded42c8b84b92564955ae348a81308b79f6b284a: Status 404 returned error can't find the container with id c8160258391651004cdafb99ded42c8b84b92564955ae348a81308b79f6b284a Nov 04 03:35:21.222828 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.222723 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" event={"ID":"568ca1fb-edf3-43af-aa30-08f3515e9bf0","Type":"ContainerStarted","Data":"81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95"} Nov 04 03:35:21.222999 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.222833 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" event={"ID":"568ca1fb-edf3-43af-aa30-08f3515e9bf0","Type":"ContainerStarted","Data":"36d21be1ba428cc0258888dae3506942c3f2f4e22802f392a7e1cbd197127ac0"} Nov 04 03:35:21.222999 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.222928 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:21.224355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.224329 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" event={"ID":"14c56253-aa80-4b21-abc7-6832d690aac8","Type":"ContainerStarted","Data":"2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83"} Nov 04 03:35:21.224498 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.224375 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" event={"ID":"14c56253-aa80-4b21-abc7-6832d690aac8","Type":"ContainerStarted","Data":"c8160258391651004cdafb99ded42c8b84b92564955ae348a81308b79f6b284a"} Nov 04 03:35:21.224498 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.224413 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:21.340130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:21.340080 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" podStartSLOduration=2.340067311 podStartE2EDuration="2.340067311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:21.338748716 +0000 UTC m=+1098.722176937" watchObservedRunningTime="2026-11-04 03:35:21.340067311 +0000 UTC m=+1098.723495376" Nov 04 03:35:24.362208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:24.362167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:28.761759 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:28.761704 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" podStartSLOduration=8.761689728 podStartE2EDuration="8.761689728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:21.47228691 +0000 UTC m=+1098.855714953" watchObservedRunningTime="2026-11-04 03:35:28.761689728 +0000 UTC m=+1106.145117779" Nov 04 03:35:28.762531 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:28.762511 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:35:29.252914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:29.252880 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:32.233777 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:32.233739 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:35:32.234225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:32.233963 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:35:34.125056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:34.125021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:37.440903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:37.440865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:38.872062 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:38.872026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:39.122278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:39.122173 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:41.180102 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:41.180068 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:41.431318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:41.431193 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:43.779863 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:43.779824 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:44.027402 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:44.027363 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:44.950811 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:44.950780 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp"] Nov 04 03:35:44.956963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:44.956944 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.011800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.011767 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp"] Nov 04 03:35:45.059207 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.059170 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bd5d7d46b00c\"" Nov 04 03:35:45.059207 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.059182 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 04 03:35:45.059435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.059268 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d2hb5\"" Nov 04 03:35:45.059435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.059344 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 04 03:35:45.060539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.060512 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 04 03:35:45.060539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.060524 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 04 03:35:45.076161 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.076135 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.076299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.076168 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn82m\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.076299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.076250 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.076299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.076288 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.177088 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.177046 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.177265 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.177101 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn82m\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.177265 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.177164 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.177265 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.177211 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.177798 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.177775 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.178014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.177988 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.180050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.180026 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.239137 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.239059 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn82m\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m\") pod \"payload-processing-external-model-6ffc4dfcb5-7l6vp\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.271806 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.271769 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:45.656501 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:45.656444 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp"] Nov 04 03:35:45.659243 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:35:45.659217 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a276d4_14cd_4615_8ea4_0525290e1103.slice/crio-b793f3bafadf4018913fae05bdd886bc9ada387b21dd7efba19caebd9a3a82d7 WatchSource:0}: Error finding container b793f3bafadf4018913fae05bdd886bc9ada387b21dd7efba19caebd9a3a82d7: Status 404 returned error can't find the container with id b793f3bafadf4018913fae05bdd886bc9ada387b21dd7efba19caebd9a3a82d7 Nov 04 03:35:46.355030 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:35:46.354992 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:35:46.373331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:46.373294 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" event={"ID":"49a276d4-14cd-4615-8ea4-0525290e1103","Type":"ContainerStarted","Data":"5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d"} Nov 04 03:35:46.373505 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:46.373352 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" event={"ID":"49a276d4-14cd-4615-8ea4-0525290e1103","Type":"ContainerStarted","Data":"b793f3bafadf4018913fae05bdd886bc9ada387b21dd7efba19caebd9a3a82d7"} Nov 04 03:35:46.373505 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:46.373449 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:46.494718 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:46.494657 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" podStartSLOduration=2.49463413 podStartE2EDuration="2.49463413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:35:46.490889263 +0000 UTC m=+1123.874317315" watchObservedRunningTime="2026-11-04 03:35:46.49463413 +0000 UTC m=+1123.878062174" Nov 04 03:35:48.736925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:48.736887 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:53.414320 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:53.414283 4959 generic.go:356] "Generic (PLEG): container finished" podID="5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" containerID="e75ec948b2ff9277ac2d66634308ce781ae55b7b3b7e7d6d4fff843b4096110a" exitCode=0 Nov 04 03:35:53.414822 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:53.414379 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" event={"ID":"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e","Type":"ContainerDied","Data":"e75ec948b2ff9277ac2d66634308ce781ae55b7b3b7e7d6d4fff843b4096110a"} Nov 04 03:35:53.414822 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:53.414583 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:53.714886 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:53.714803 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:54.571095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.571071 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:35:54.694787 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.694701 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path\") pod \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " Nov 04 03:35:54.694941 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.694801 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle\") pod \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " Nov 04 03:35:54.694941 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.694839 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle\") pod \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " Nov 04 03:35:54.695070 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695039 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf\" (UniqueName: \"kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf\") pod \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\" (UID: \"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e\") " Nov 04 03:35:54.695130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695081 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path" pod "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" (UID: "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:35:54.695272 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695245 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle" pod "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" (UID: "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:35:54.695339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695280 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle" pod "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" (UID: "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:35:54.695649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695624 4959 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-archives-path\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:54.695755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695653 4959 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-trusted-ca-bundle\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:54.695755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.695668 4959 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-service-ca-bundle\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:54.697459 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.697433 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf" pod "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" (UID: "5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e"). InnerVolumeSpecName "kube-api-access-fh8sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:35:54.796421 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:54.796386 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8sf\" (UniqueName: \"kubernetes.io/projected/5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e-kube-api-access-fh8sf\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:35:55.427336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:55.427309 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" Nov 04 03:35:55.427556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:55.427306 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x2dlh-6xbjc" event={"ID":"5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e","Type":"ContainerDied","Data":"73a470080e58e3726170956de18b3273419641f0d04de012a098120456f5c234"} Nov 04 03:35:55.427556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:55.427427 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73a470080e58e3726170956de18b3273419641f0d04de012a098120456f5c234" Nov 04 03:35:56.591218 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:56.591180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:56.856559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:56.856437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:57.106179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:57.106145 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:35:57.382556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:57.382526 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:35:58.496131 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:58.496096 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:35:58.981508 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:35:58.981332 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:03.305788 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:03.305732 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:03.552228 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:03.552183 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:03.810313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:03.810218 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:07.567880 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.567846 4959 generic.go:356] "Generic (PLEG): container finished" podID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerID="3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531" exitCode=0 Nov 04 03:36:07.568356 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.567921 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" event={"ID":"50b5a408-6a57-436d-a4c7-10faa9e27fda","Type":"ContainerDied","Data":"3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531"} Nov 04 03:36:07.568419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.568388 4959 scope.go:122] "RemoveContainer" containerID="3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531" Nov 04 03:36:07.569804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.569782 4959 generic.go:356] "Generic (PLEG): container finished" podID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerID="2611813c2acbae393b0b64416cf8e807dc41775ee540d47a63adb5e7b973d584" exitCode=0 Nov 04 03:36:07.569903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.569851 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" event={"ID":"655034e2-d0d9-475d-b1b5-e5a412eb9bc5","Type":"ContainerDied","Data":"2611813c2acbae393b0b64416cf8e807dc41775ee540d47a63adb5e7b973d584"} Nov 04 03:36:07.569976 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.569952 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:07.570031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.570019 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:07.571975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.570439 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:07.571975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.570627 4959 scope.go:122] "RemoveContainer" containerID="2611813c2acbae393b0b64416cf8e807dc41775ee540d47a63adb5e7b973d584" Nov 04 03:36:07.579814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.579782 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:07.579912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.579863 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:07.579964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.579912 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:07.702117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.702067 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br"] Nov 04 03:36:07.703157 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.703129 4959 cpu_manager.go:414] "containerMap: removing container" podUID="5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" containerName="insights-gathering" Nov 04 03:36:07.703157 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.703156 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" containerName="insights-gathering" Nov 04 03:36:07.703411 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.703272 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bccf5c7-ef99-4a3c-b5a4-40a6fae6eb8e" containerName="insights-gathering" Nov 04 03:36:07.705900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.705877 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.803406 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.803371 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br"] Nov 04 03:36:07.816696 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816668 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.816843 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816713 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmm8f\" (UniqueName: \"kubernetes.io/projected/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-kube-api-access-bmm8f\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.816843 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816792 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.816931 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816863 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.816931 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816915 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.817025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816936 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.817025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.816984 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.817180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.817159 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.817246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.817198 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918406 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918440 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918439 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918456 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918630 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918664 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918739 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918762 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmm8f\" (UniqueName: \"kubernetes.io/projected/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-kube-api-access-bmm8f\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918820 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.918885 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.918875 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.919518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.919204 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.919518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.919375 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.919518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.919459 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.919800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.919667 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.919800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.919760 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.921369 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.921344 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.921702 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.921683 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.985384 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.982374 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmm8f\" (UniqueName: \"kubernetes.io/projected/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-kube-api-access-bmm8f\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:07.985384 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:07.983822 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6s4br\" (UID: \"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:08.025622 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.025583 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:08.343083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.343046 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:08.521501 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.502391 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br"] Nov 04 03:36:08.521501 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.516453 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:08.521501 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.516567 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:08.521501 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.516607 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:08.593543 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.590584 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" event={"ID":"655034e2-d0d9-475d-b1b5-e5a412eb9bc5","Type":"ContainerStarted","Data":"cc77707d821e4c2202bd39199796a40701c1929ef6adbc24fbf9eae900bb786b"} Nov 04 03:36:08.593543 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.593415 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" event={"ID":"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7","Type":"ContainerStarted","Data":"0d5d81e5573b083e77b71dcf4b6f02749ed2298db9273459852a36088c5351b9"} Nov 04 03:36:08.614494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.613246 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" event={"ID":"50b5a408-6a57-436d-a4c7-10faa9e27fda","Type":"ContainerStarted","Data":"262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b"} Nov 04 03:36:08.614663 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.614507 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:36:08.626495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.622656 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 04 03:36:08.626495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:08.622710 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 04 03:36:09.261862 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.261823 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:36:09.263031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.263000 4959 patch_prober.go:29] interesting pod/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 03:36:09.263201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.263050 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 03:36:09.610049 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.610011 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 04 03:36:09.610736 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.610684 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 04 03:36:09.654574 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.647149 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" event={"ID":"6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7","Type":"ContainerStarted","Data":"10a2f4d5dab9a1676c782b8268d730945c0cc2de61935f6ad97f6b514f38acf7"} Nov 04 03:36:09.786497 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:09.781201 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podStartSLOduration=2.781183485 podStartE2EDuration="2.781183485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:09.778428112 +0000 UTC m=+1147.161856152" watchObservedRunningTime="2026-11-04 03:36:09.781183485 +0000 UTC m=+1147.164611527" Nov 04 03:36:10.029552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.028706 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:10.029552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.028873 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:10.029552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.028920 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:10.261844 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.261807 4959 patch_prober.go:29] interesting pod/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 03:36:10.262045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.261872 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 03:36:10.604930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.604893 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 04 03:36:10.605224 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:10.605196 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 04 03:36:11.027890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:11.027802 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:11.028442 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:11.028411 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:11.261029 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:11.260989 4959 patch_prober.go:29] interesting pod/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 03:36:11.261204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:11.261042 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 03:36:11.610497 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:11.610428 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 04 03:36:11.610701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:11.610519 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 04 03:36:12.027557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.026631 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:12.027557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.026693 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:12.047598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.047563 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:36:12.261766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.261729 4959 patch_prober.go:29] interesting pod/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 03:36:12.261978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.261779 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 03:36:12.605875 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.605836 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 04 03:36:12.606139 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:12.606113 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 04 03:36:13.027955 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.027859 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:13.028220 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.028188 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:13.058516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.058482 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:36:13.226977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.226933 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp"] Nov 04 03:36:13.265678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.265568 4959 patch_prober.go:29] interesting pod/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 04 03:36:13.265678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.265620 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 04 03:36:13.465405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.465367 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:13.609500 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.607867 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Nov 04 03:36:13.609500 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:13.607928 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Nov 04 03:36:14.027095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:14.026971 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:14.027095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:14.027053 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:14.268206 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:14.268177 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:36:14.269099 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:14.269077 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:36:14.609419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:14.609394 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:36:14.610694 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:14.610671 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:36:15.026311 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:15.026222 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:15.026311 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:15.026277 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:16.026276 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:16.026243 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6s4br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Nov 04 03:36:16.026683 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:16.026294 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" podUID="6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Nov 04 03:36:16.355871 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:36:16.355839 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:36:17.746756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:17.746723 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:17.748124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:17.748083 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:17.749096 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:17.749071 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6s4br" Nov 04 03:36:18.227170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.227131 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:18.256558 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.256495 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" podUID="49a276d4-14cd-4615-8ea4-0525290e1103" containerName="payload-processing" containerID="cri-o://5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d" gracePeriod=25 Nov 04 03:36:18.526728 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.526706 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:36:18.661492 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.661441 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m\") pod \"49a276d4-14cd-4615-8ea4-0525290e1103\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " Nov 04 03:36:18.661638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.661612 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume\") pod \"49a276d4-14cd-4615-8ea4-0525290e1103\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " Nov 04 03:36:18.661690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.661666 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume\") pod \"49a276d4-14cd-4615-8ea4-0525290e1103\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " Nov 04 03:36:18.661726 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.661693 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials\") pod \"49a276d4-14cd-4615-8ea4-0525290e1103\" (UID: \"49a276d4-14cd-4615-8ea4-0525290e1103\") " Nov 04 03:36:18.662030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.662002 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume" pod "49a276d4-14cd-4615-8ea4-0525290e1103" (UID: "49a276d4-14cd-4615-8ea4-0525290e1103"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:18.662165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.662114 4959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-routing-overlay-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:18.662165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.662119 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume" pod "49a276d4-14cd-4615-8ea4-0525290e1103" (UID: "49a276d4-14cd-4615-8ea4-0525290e1103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:18.664353 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.664325 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials" pod "49a276d4-14cd-4615-8ea4-0525290e1103" (UID: "49a276d4-14cd-4615-8ea4-0525290e1103"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:18.664457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.664406 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m" pod "49a276d4-14cd-4615-8ea4-0525290e1103" (UID: "49a276d4-14cd-4615-8ea4-0525290e1103"). InnerVolumeSpecName "kube-api-access-xn82m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:18.735154 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.735127 4959 generic.go:356] "Generic (PLEG): container finished" podID="49a276d4-14cd-4615-8ea4-0525290e1103" containerID="5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d" exitCode=0 Nov 04 03:36:18.735290 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.735174 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" Nov 04 03:36:18.735290 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.735200 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" event={"ID":"49a276d4-14cd-4615-8ea4-0525290e1103","Type":"ContainerDied","Data":"5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d"} Nov 04 03:36:18.735290 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.735261 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp" event={"ID":"49a276d4-14cd-4615-8ea4-0525290e1103","Type":"ContainerDied","Data":"b793f3bafadf4018913fae05bdd886bc9ada387b21dd7efba19caebd9a3a82d7"} Nov 04 03:36:18.735457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.735288 4959 scope.go:122] "RemoveContainer" containerID="5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d" Nov 04 03:36:18.744855 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.744846 4959 scope.go:122] "RemoveContainer" containerID="5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d" Nov 04 03:36:18.745108 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:36:18.745089 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d\": container with ID starting with 5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d not found: ID does not exist" containerID="5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d" Nov 04 03:36:18.745171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.745115 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d"} err="failed to get container status \"5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d\": rpc error: code = NotFound desc = could not find container \"5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d\": container with ID starting with 5aded57777c1721c71cee2e28c63d480f136ba149e52017c50d954f783c9989d not found: ID does not exist" Nov 04 03:36:18.762826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.762755 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn82m\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-kube-api-access-xn82m\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:18.762826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.762782 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49a276d4-14cd-4615-8ea4-0525290e1103-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:18.762826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.762792 4959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49a276d4-14cd-4615-8ea4-0525290e1103-provider-credentials\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:18.902312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.902285 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp"] Nov 04 03:36:18.904662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:18.904643 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ffc4dfcb5-7l6vp"] Nov 04 03:36:19.299319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:19.299283 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:36:20.801025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:20.800991 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a276d4-14cd-4615-8ea4-0525290e1103" path="/var/lib/kubelet/pods/49a276d4-14cd-4615-8ea4-0525290e1103/volumes" Nov 04 03:36:22.842975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:22.842938 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm"] Nov 04 03:36:22.843848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:22.843826 4959 cpu_manager.go:414] "containerMap: removing container" podUID="49a276d4-14cd-4615-8ea4-0525290e1103" containerName="payload-processing" Nov 04 03:36:22.843900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:22.843852 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a276d4-14cd-4615-8ea4-0525290e1103" containerName="payload-processing" Nov 04 03:36:22.844011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:22.843997 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a276d4-14cd-4615-8ea4-0525290e1103" containerName="payload-processing" Nov 04 03:36:22.849305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:22.849281 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:22.969415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:22.969383 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm"] Nov 04 03:36:23.019613 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.019576 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.019791 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.019623 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbzmw\" (UniqueName: \"kubernetes.io/projected/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-kube-api-access-fbzmw\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.019791 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.019698 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.019876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.019817 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.019876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.019868 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.019969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.019900 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.020045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.020028 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.020103 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.020072 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.020145 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.020101 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121048 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121125 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121155 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121204 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121229 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121243 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121273 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121314 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbzmw\" (UniqueName: \"kubernetes.io/projected/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-kube-api-access-fbzmw\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121691 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121342 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121691 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121520 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121747 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121787 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.121906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.121856 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.122162 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.122098 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.123562 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.123538 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.123791 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.123774 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.184233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.184198 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.184372 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.184212 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbzmw\" (UniqueName: \"kubernetes.io/projected/7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd-kube-api-access-fbzmw\") pod \"maas-default-gateway-openshift-default-6dcf757899-hj5bm\" (UID: \"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.316268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.316226 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:23.463198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.463119 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:23.890616 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.890591 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm"] Nov 04 03:36:23.892494 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:23.892452 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6a2e5a_d8b8_4c77_a9dc_d55f318f58dd.slice/crio-bb90b948c43d9ecaf37c14fd458a5ecab32c7517ea4a0bf8941ce37ba1a9f20e WatchSource:0}: Error finding container bb90b948c43d9ecaf37c14fd458a5ecab32c7517ea4a0bf8941ce37ba1a9f20e: Status 404 returned error can't find the container with id bb90b948c43d9ecaf37c14fd458a5ecab32c7517ea4a0bf8941ce37ba1a9f20e Nov 04 03:36:23.894920 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.894893 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:23.894994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.894954 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:23.894994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:23.894981 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:24.774618 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:24.774583 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" event={"ID":"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd","Type":"ContainerStarted","Data":"e722f908e618c0869d7454c8892e7eaa9ca345906ee77baaac710e224443bd93"} Nov 04 03:36:24.774793 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:24.774636 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" event={"ID":"7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd","Type":"ContainerStarted","Data":"bb90b948c43d9ecaf37c14fd458a5ecab32c7517ea4a0bf8941ce37ba1a9f20e"} Nov 04 03:36:24.811721 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:24.811679 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf"] Nov 04 03:36:24.811957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:24.811928 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" containerID="cri-o://080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8" gracePeriod=30 Nov 04 03:36:24.920243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:24.920193 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podStartSLOduration=2.920180186 podStartE2EDuration="2.920180186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:24.919433302 +0000 UTC m=+1162.302861345" watchObservedRunningTime="2026-11-04 03:36:24.920180186 +0000 UTC m=+1162.303608227" Nov 04 03:36:25.463519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:25.463454 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:25.465336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:25.465307 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:25.465460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:25.465361 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:26.463763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:26.463731 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:26.464149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:26.463783 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:27.464390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:27.464354 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:27.464790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:27.464451 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:28.417096 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:28.417049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:28.464175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:28.464144 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:28.464339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:28.464200 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:29.463455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:29.463416 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:29.463957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:29.463517 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:30.095804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.095740 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:36:30.222582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222540 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.222778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222597 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.222778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222619 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.222778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222640 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.222778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222706 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.222778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222774 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.223056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222831 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.223056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222861 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:30.223056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.222978 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.223056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223019 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket\") pod \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\" (UID: \"3d2e8ed1-88f7-4235-9ea1-c731665829b2\") " Nov 04 03:36:30.223266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223085 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:30.223327 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223290 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:30.223386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223358 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:30.223536 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223503 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:30.223536 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223523 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.223744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223547 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.223744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223562 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.223744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.223574 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.225414 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.225389 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "kube-api-access-csf42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:30.225530 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.225457 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:36:30.225644 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.225627 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:36:30.225770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.225755 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token" pod "3d2e8ed1-88f7-4235-9ea1-c731665829b2" (UID: "3d2e8ed1-88f7-4235-9ea1-c731665829b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:30.325027 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.324994 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.325027 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.325026 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csf42\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-kube-api-access-csf42\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.325027 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.325036 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2e8ed1-88f7-4235-9ea1-c731665829b2-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.325269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.325052 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.325269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.325067 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2e8ed1-88f7-4235-9ea1-c731665829b2-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:30.463539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.463437 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:30.463539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.463508 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:30.850053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.850017 4959 generic.go:356] "Generic (PLEG): container finished" podID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerID="080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8" exitCode=0 Nov 04 03:36:30.850210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.850078 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" event={"ID":"3d2e8ed1-88f7-4235-9ea1-c731665829b2","Type":"ContainerDied","Data":"080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8"} Nov 04 03:36:30.850210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.850084 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" Nov 04 03:36:30.850210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.850131 4959 scope.go:122] "RemoveContainer" containerID="080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8" Nov 04 03:36:30.850346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.850121 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf" event={"ID":"3d2e8ed1-88f7-4235-9ea1-c731665829b2","Type":"ContainerDied","Data":"3e9e378055d5dab215e5456bea2de845b5d56ccf9ffdc7ce9f4789877dc84dc4"} Nov 04 03:36:30.862739 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.862716 4959 scope.go:122] "RemoveContainer" containerID="fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc" Nov 04 03:36:30.871879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.871856 4959 scope.go:122] "RemoveContainer" containerID="080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8" Nov 04 03:36:30.872148 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:36:30.872118 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8\": container with ID starting with 080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8 not found: ID does not exist" containerID="080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8" Nov 04 03:36:30.872203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.872156 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8"} err="failed to get container status \"080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8\": rpc error: code = NotFound desc = could not find container \"080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8\": container with ID starting with 080062136d7bad5353040ffaf1c2dffbc93be2bb8723824837ee4bf648ba17b8 not found: ID does not exist" Nov 04 03:36:30.872203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.872174 4959 scope.go:122] "RemoveContainer" containerID="fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc" Nov 04 03:36:30.872403 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:36:30.872388 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc\": container with ID starting with fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc not found: ID does not exist" containerID="fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc" Nov 04 03:36:30.872449 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:30.872411 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc"} err="failed to get container status \"fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc\": rpc error: code = NotFound desc = could not find container \"fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc\": container with ID starting with fb444ad13dad1edee02761fe19de8a7868aa03767fccc06d1fe6fa3076ba8bfc not found: ID does not exist" Nov 04 03:36:31.017353 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.017310 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf"] Nov 04 03:36:31.021259 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.021227 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-cxnbf"] Nov 04 03:36:31.302800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.302766 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm"] Nov 04 03:36:31.303307 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.303294 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" Nov 04 03:36:31.303358 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.303308 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" Nov 04 03:36:31.303358 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.303322 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" Nov 04 03:36:31.303358 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.303327 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" Nov 04 03:36:31.303483 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.303407 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" Nov 04 03:36:31.303483 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.303415 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" containerName="istio-proxy" Nov 04 03:36:31.308321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.308299 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335386 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335427 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335532 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qq6\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335579 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335613 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335639 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335782 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335832 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.335940 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.335922 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.370001 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.369970 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm"] Nov 04 03:36:31.402415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.402386 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n"] Nov 04 03:36:31.406596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.406462 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.437173 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437134 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437212 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437249 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437346 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437397 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437429 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qq6\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437506 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437893 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437551 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.437893 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.437590 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.438087 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.438062 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.438826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.438804 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.439119 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.439075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.440127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.440103 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.440379 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.440358 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.444080 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.444057 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.456598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.456572 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.465215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.465186 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n"] Nov 04 03:36:31.468521 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.468054 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hj5bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 04 03:36:31.468521 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.468108 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" podUID="7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 04 03:36:31.470000 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.469980 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:36:31.470124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.470105 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:36:31.491563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.487892 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-491fe7-openshift-default-dockercfg-fpm7q\"" Nov 04 03:36:31.491563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.487973 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:36:31.491563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.488077 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:36:31.530627 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.530589 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.530867 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.530843 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qq6\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.538994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.538972 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.539078 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.539043 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgztw\" (UniqueName: \"kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.539133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.539119 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.622015 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.621984 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:31.640882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.640853 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgztw\" (UniqueName: \"kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.640989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.640915 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.641087 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.641071 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.641934 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.641897 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.643782 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.643755 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.704179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.704147 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgztw\" (UniqueName: \"kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw\") pod \"maas-api-revoke-sub-83f29174-l4c8n\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:31.744190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:31.744163 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:32.067399 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:32.067357 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a0c4e5_099d_42ca_ab8e_59faea597b3f.slice/crio-4eaa792fd29bbc80e2857146436878d508d3a42f925472c589e8fb4d2b1c9489 WatchSource:0}: Error finding container 4eaa792fd29bbc80e2857146436878d508d3a42f925472c589e8fb4d2b1c9489: Status 404 returned error can't find the container with id 4eaa792fd29bbc80e2857146436878d508d3a42f925472c589e8fb4d2b1c9489 Nov 04 03:36:32.068255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.068223 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm"] Nov 04 03:36:32.069878 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.069848 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:32.069966 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.069917 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:32.069966 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.069947 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:32.167306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.167267 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n"] Nov 04 03:36:32.170763 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:32.170735 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ddcd5b_2dab_4171_9dbd_6029bb4b9ff0.slice/crio-a3f84e344935badb905200a181efd32ee8ddaa565f81cdd8bee715c95cce78b5 WatchSource:0}: Error finding container a3f84e344935badb905200a181efd32ee8ddaa565f81cdd8bee715c95cce78b5: Status 404 returned error can't find the container with id a3f84e344935badb905200a181efd32ee8ddaa565f81cdd8bee715c95cce78b5 Nov 04 03:36:32.801824 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.801742 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2e8ed1-88f7-4235-9ea1-c731665829b2" path="/var/lib/kubelet/pods/3d2e8ed1-88f7-4235-9ea1-c731665829b2/volumes" Nov 04 03:36:32.867199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.867163 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" event={"ID":"a2a0c4e5-099d-42ca-ab8e-59faea597b3f","Type":"ContainerStarted","Data":"dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2"} Nov 04 03:36:32.867388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.867230 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" event={"ID":"a2a0c4e5-099d-42ca-ab8e-59faea597b3f","Type":"ContainerStarted","Data":"4eaa792fd29bbc80e2857146436878d508d3a42f925472c589e8fb4d2b1c9489"} Nov 04 03:36:32.869104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.869073 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" event={"ID":"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0","Type":"ContainerStarted","Data":"4e051b9e75187d8ed1193aec4d7afd87a04ce7752068089e58601ea40fe31fe8"} Nov 04 03:36:32.869252 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.869126 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" event={"ID":"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0","Type":"ContainerStarted","Data":"a3f84e344935badb905200a181efd32ee8ddaa565f81cdd8bee715c95cce78b5"} Nov 04 03:36:32.872230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.872206 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:32.872903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.872883 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:32.874127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.874104 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hj5bm" Nov 04 03:36:32.993812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:32.993764 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podStartSLOduration=1.993750396 podStartE2EDuration="1.993750396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:32.991966658 +0000 UTC m=+1170.375394700" watchObservedRunningTime="2026-11-04 03:36:32.993750396 +0000 UTC m=+1170.377178502" Nov 04 03:36:33.111964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.111874 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" podStartSLOduration=2.111858023 podStartE2EDuration="2.111858023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:33.1086996 +0000 UTC m=+1170.492127641" watchObservedRunningTime="2026-11-04 03:36:33.111858023 +0000 UTC m=+1170.495286100" Nov 04 03:36:33.338996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.338961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:33.622386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.622308 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:33.624261 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.624235 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 03:36:33.624373 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.624297 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 03:36:33.876702 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.876615 4959 generic.go:356] "Generic (PLEG): container finished" podID="c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" containerID="4e051b9e75187d8ed1193aec4d7afd87a04ce7752068089e58601ea40fe31fe8" exitCode=0 Nov 04 03:36:33.877083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:33.876694 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" event={"ID":"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0","Type":"ContainerDied","Data":"4e051b9e75187d8ed1193aec4d7afd87a04ce7752068089e58601ea40fe31fe8"} Nov 04 03:36:34.622874 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:34.622841 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 03:36:34.623067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:34.622904 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 03:36:35.075555 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.075524 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:35.211512 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.211398 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca\") pod \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " Nov 04 03:36:35.211512 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.211500 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca\") pod \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " Nov 04 03:36:35.211828 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.211627 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw\" (UniqueName: \"kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw\") pod \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\" (UID: \"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0\") " Nov 04 03:36:35.211828 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.211754 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca" pod "c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" (UID: "c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:36:35.212143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.212119 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.214605 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.214578 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw" pod "c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" (UID: "c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0"). InnerVolumeSpecName "kube-api-access-jgztw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:36:35.214721 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.214693 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca" pod "c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" (UID: "c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:36:35.313670 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.313637 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgztw\" (UniqueName: \"kubernetes.io/projected/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-kube-api-access-jgztw\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.313670 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.313669 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:36:35.623082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.623045 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 03:36:35.623254 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.623108 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 03:36:35.914646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.914563 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" Nov 04 03:36:35.914792 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.914563 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n" event={"ID":"c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0","Type":"ContainerDied","Data":"a3f84e344935badb905200a181efd32ee8ddaa565f81cdd8bee715c95cce78b5"} Nov 04 03:36:35.914792 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:35.914695 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f84e344935badb905200a181efd32ee8ddaa565f81cdd8bee715c95cce78b5" Nov 04 03:36:36.623132 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:36.623097 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 03:36:36.623559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:36.623159 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 03:36:37.623184 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:37.623141 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 03:36:37.623739 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:37.623213 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 03:36:38.260160 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:38.255843 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:38.622903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:38.622864 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 04 03:36:38.623068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:38.622930 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 04 03:36:39.627221 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:39.627190 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:39.627717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:39.627652 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:39.628322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:39.628306 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:36:41.013052 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:41.013013 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:42.348452 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:42.348414 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:43.283050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.283013 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p"] Nov 04 03:36:43.283677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.283658 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" containerName="revoke-keys" Nov 04 03:36:43.283741 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.283681 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" containerName="revoke-keys" Nov 04 03:36:43.283812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.283802 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" containerName="revoke-keys" Nov 04 03:36:43.287958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.287937 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.341613 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.341582 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5d459a-openshift-default-dockercfg-5rj77\"" Nov 04 03:36:43.393912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.393884 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p"] Nov 04 03:36:43.415036 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415001 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415048 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415077 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415197 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415236 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415263 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415497 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415355 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415534 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.415645 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.415626 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm587\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.496495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.466123 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:43.516809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.516776 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.516994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.516832 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm587\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.516994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.516908 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.516994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.516934 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.516994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.516960 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.516994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.516989 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517014 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517036 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517074 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517165 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517656 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517779 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517687 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517779 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517707 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.517902 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.517807 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.519590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.519562 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.519924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.519907 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.580116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.580085 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.580300 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.580184 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm587\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587\") pod \"e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.615498 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.607718 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:43.756619 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:43.756585 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:44.055755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.055725 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p"] Nov 04 03:36:44.057259 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:44.057221 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df7a391_1ace_4782_8f2c_410d14c4429e.slice/crio-69104b54f1cc3d39fe630b73fa48a6b78ffe083351d4f48812e6ff71cdad3d3f WatchSource:0}: Error finding container 69104b54f1cc3d39fe630b73fa48a6b78ffe083351d4f48812e6ff71cdad3d3f: Status 404 returned error can't find the container with id 69104b54f1cc3d39fe630b73fa48a6b78ffe083351d4f48812e6ff71cdad3d3f Nov 04 03:36:44.059979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.059948 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:44.060094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.060010 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:44.060094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.060039 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:36:44.755090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.755057 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:44.974428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.974384 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" event={"ID":"6df7a391-1ace-4782-8f2c-410d14c4429e","Type":"ContainerStarted","Data":"6b9e28695899ba0892316b1aba344e5b3e429e0f1bd26ae5d6cb6cc6ec282fb6"} Nov 04 03:36:44.974651 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:44.974450 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" event={"ID":"6df7a391-1ace-4782-8f2c-410d14c4429e","Type":"ContainerStarted","Data":"69104b54f1cc3d39fe630b73fa48a6b78ffe083351d4f48812e6ff71cdad3d3f"} Nov 04 03:36:45.002784 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:45.002707 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:45.097145 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:45.097094 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podStartSLOduration=2.097080642 podStartE2EDuration="2.097080642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:45.095594559 +0000 UTC m=+1182.479022630" watchObservedRunningTime="2026-11-04 03:36:45.097080642 +0000 UTC m=+1182.480508682" Nov 04 03:36:45.610258 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:45.610222 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:45.615960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:45.615934 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:45.980647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:45.980571 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:45.981560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:45.981538 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:36:46.354838 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:36:46.354803 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:36:47.337231 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:47.337197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:48.214317 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.214278 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:48.298741 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.298704 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-859b946898-skctn"] Nov 04 03:36:48.302681 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.302659 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.358126 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.358084 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 04 03:36:48.358126 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.358118 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f2c682677175\"" Nov 04 03:36:48.393980 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.393950 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.394137 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.394111 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpmt\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.394194 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.394147 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.394194 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.394180 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.408308 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.408283 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 04 03:36:48.408308 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.408308 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 04 03:36:48.409102 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.409085 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d2hb5\"" Nov 04 03:36:48.409102 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.409097 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 04 03:36:48.412120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.412097 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-859b946898-skctn"] Nov 04 03:36:48.474034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.473956 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:48.495331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.495302 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.495516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.495461 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpmt\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.495649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.495544 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.495649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.495576 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.496203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.496185 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.496325 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.496307 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.498152 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.498129 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.556774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.556743 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpmt\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt\") pod \"payload-processing-external-model-859b946898-skctn\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.619932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.618634 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:48.786978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:48.786950 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:49.042447 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:49.042360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:49.060511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:49.060402 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-859b946898-skctn"] Nov 04 03:36:49.063089 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:49.063060 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5ca16b_7ce8_4c31_a15b_f3273000c867.slice/crio-a53f9da3532227842cd197a0a53685805a3eee0c8b4cecae47ccc3aff54f4234 WatchSource:0}: Error finding container a53f9da3532227842cd197a0a53685805a3eee0c8b4cecae47ccc3aff54f4234: Status 404 returned error can't find the container with id a53f9da3532227842cd197a0a53685805a3eee0c8b4cecae47ccc3aff54f4234 Nov 04 03:36:49.298834 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:49.298754 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:49.883193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:49.883153 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:50.039975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:50.039927 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" event={"ID":"9c5ca16b-7ce8-4c31-a15b-f3273000c867","Type":"ContainerStarted","Data":"9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e"} Nov 04 03:36:50.040174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:50.040005 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" event={"ID":"9c5ca16b-7ce8-4c31-a15b-f3273000c867","Type":"ContainerStarted","Data":"a53f9da3532227842cd197a0a53685805a3eee0c8b4cecae47ccc3aff54f4234"} Nov 04 03:36:50.040760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:50.040738 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:36:50.136430 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:50.136344 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:50.190496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:50.180462 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podStartSLOduration=2.180440895 podStartE2EDuration="2.180440895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:50.178005807 +0000 UTC m=+1187.561433851" watchObservedRunningTime="2026-11-04 03:36:50.180440895 +0000 UTC m=+1187.563868936" Nov 04 03:36:51.516982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:51.516927 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:51.823050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:51.822957 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:53.900012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:53.899983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:54.194344 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:54.194253 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:54.948537 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:54.948495 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v"] Nov 04 03:36:54.952766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:54.952750 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.033845 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.033802 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.034029 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.033976 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.034073 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.034034 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.034193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.034165 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv47n\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.058607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.058565 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d1e2ab\"/\"routing-overlay\"" Nov 04 03:36:55.058607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.058583 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d1e2ab\"/\"kube-root-ca.crt\"" Nov 04 03:36:55.058607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.058616 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d1e2ab\"/\"agc-cred-451b60bc8666\"" Nov 04 03:36:55.058872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.058611 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d1e2ab\"/\"openshift-service-ca.crt\"" Nov 04 03:36:55.058872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.058623 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d1e2ab\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d1e2ab\"" Nov 04 03:36:55.058872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.058709 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d1e2ab\"/\"payload-processing-external-model-e2e--6660102d-dockercfg-v4pdc\"" Nov 04 03:36:55.061451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.061428 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v"] Nov 04 03:36:55.135243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.135202 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.135443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.135275 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.135443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.135313 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv47n\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.135443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.135348 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.135978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.135950 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.136247 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.136013 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.137941 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.137920 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.197207 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.197169 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv47n\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n\") pod \"payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.267099 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.266992 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:55.647121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:55.647079 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v"] Nov 04 03:36:55.648559 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:55.648527 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2950b8be_019d_4e7f_974c_ca4b9d64a404.slice/crio-7d931b822f536a11a820349ad7a13fd2a24f8619afa0f98ab5621a07973a4755 WatchSource:0}: Error finding container 7d931b822f536a11a820349ad7a13fd2a24f8619afa0f98ab5621a07973a4755: Status 404 returned error can't find the container with id 7d931b822f536a11a820349ad7a13fd2a24f8619afa0f98ab5621a07973a4755 Nov 04 03:36:56.085023 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:56.084991 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" event={"ID":"2950b8be-019d-4e7f-974c-ca4b9d64a404","Type":"ContainerStarted","Data":"9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697"} Nov 04 03:36:56.085460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:56.085044 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" event={"ID":"2950b8be-019d-4e7f-974c-ca4b9d64a404","Type":"ContainerStarted","Data":"7d931b822f536a11a820349ad7a13fd2a24f8619afa0f98ab5621a07973a4755"} Nov 04 03:36:56.085460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:56.085142 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:36:56.204443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:56.204381 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" podStartSLOduration=2.204364844 podStartE2EDuration="2.204364844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:56.202277452 +0000 UTC m=+1193.585705493" watchObservedRunningTime="2026-11-04 03:36:56.204364844 +0000 UTC m=+1193.587792890" Nov 04 03:36:56.524865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:56.524767 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:56.875539 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:56.873568 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:36:57.015557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.015521 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn"] Nov 04 03:36:57.019624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.019609 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.070434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.070391 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7v6x\" (UniqueName: \"kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.070642 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.070608 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.070695 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.070656 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.079290 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.079264 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:36:57.087487 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.084083 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:36:57.087487 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.084704 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:36:57.087487 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.086783 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn"] Nov 04 03:36:57.124491 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.124439 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:36:57.172179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.172089 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7v6x\" (UniqueName: \"kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.172344 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.172186 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.172344 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.172211 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.173081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.173054 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.174891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.174869 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.235051 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.235016 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7v6x\" (UniqueName: \"kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x\") pod \"maas-api-revoke-sub-653cbc1a-8fvtn\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.334396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.334356 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:36:57.730614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:57.730521 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn"] Nov 04 03:36:57.734036 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:36:57.734010 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05af6cd7_a401_4ed8_a220_3d4e45480d1c.slice/crio-18dc7bb1731172a015c693477db3aa2a46f9f2f8683e84c1c90125af46cefdce WatchSource:0}: Error finding container 18dc7bb1731172a015c693477db3aa2a46f9f2f8683e84c1c90125af46cefdce: Status 404 returned error can't find the container with id 18dc7bb1731172a015c693477db3aa2a46f9f2f8683e84c1c90125af46cefdce Nov 04 03:36:58.111115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:58.111076 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" event={"ID":"05af6cd7-a401-4ed8-a220-3d4e45480d1c","Type":"ContainerStarted","Data":"8e92c40f3f117b784404dbc0dcbca443736c5a7f6061368a19d3a6c8d339a687"} Nov 04 03:36:58.111675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:58.111141 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" event={"ID":"05af6cd7-a401-4ed8-a220-3d4e45480d1c","Type":"ContainerStarted","Data":"18dc7bb1731172a015c693477db3aa2a46f9f2f8683e84c1c90125af46cefdce"} Nov 04 03:36:58.231035 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:58.230976 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" podStartSLOduration=2.2309624550000002 podStartE2EDuration="2.230962455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:36:58.230097121 +0000 UTC m=+1195.613525160" watchObservedRunningTime="2026-11-04 03:36:58.230962455 +0000 UTC m=+1195.614390496" Nov 04 03:36:59.071552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:59.070529 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:36:59.170864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:59.170827 4959 generic.go:356] "Generic (PLEG): container finished" podID="05af6cd7-a401-4ed8-a220-3d4e45480d1c" containerID="8e92c40f3f117b784404dbc0dcbca443736c5a7f6061368a19d3a6c8d339a687" exitCode=0 Nov 04 03:36:59.171278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:59.170911 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" event={"ID":"05af6cd7-a401-4ed8-a220-3d4e45480d1c","Type":"ContainerDied","Data":"8e92c40f3f117b784404dbc0dcbca443736c5a7f6061368a19d3a6c8d339a687"} Nov 04 03:36:59.414607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:36:59.414543 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:00.613833 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.607001 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:37:00.762163 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.762125 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x\" (UniqueName: \"kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x\") pod \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " Nov 04 03:37:00.762359 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.762332 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca\") pod \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " Nov 04 03:37:00.762429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.762408 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca\") pod \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\" (UID: \"05af6cd7-a401-4ed8-a220-3d4e45480d1c\") " Nov 04 03:37:00.762765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.762730 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca" pod "05af6cd7-a401-4ed8-a220-3d4e45480d1c" (UID: "05af6cd7-a401-4ed8-a220-3d4e45480d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:37:00.762881 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.762845 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:00.765180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.765148 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca" pod "05af6cd7-a401-4ed8-a220-3d4e45480d1c" (UID: "05af6cd7-a401-4ed8-a220-3d4e45480d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:37:00.765590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.765564 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x" pod "05af6cd7-a401-4ed8-a220-3d4e45480d1c" (UID: "05af6cd7-a401-4ed8-a220-3d4e45480d1c"). InnerVolumeSpecName "kube-api-access-l7v6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:37:00.863924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.863892 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05af6cd7-a401-4ed8-a220-3d4e45480d1c-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:00.864120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:00.863932 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7v6x\" (UniqueName: \"kubernetes.io/projected/05af6cd7-a401-4ed8-a220-3d4e45480d1c-kube-api-access-l7v6x\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:01.048628 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:01.048591 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:37:01.186964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:01.186880 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" Nov 04 03:37:01.186964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:01.186885 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn" event={"ID":"05af6cd7-a401-4ed8-a220-3d4e45480d1c","Type":"ContainerDied","Data":"18dc7bb1731172a015c693477db3aa2a46f9f2f8683e84c1c90125af46cefdce"} Nov 04 03:37:01.186964 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:01.186928 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18dc7bb1731172a015c693477db3aa2a46f9f2f8683e84c1c90125af46cefdce" Nov 04 03:37:01.671130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:01.671084 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:02.902647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:02.902620 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:37:02.905177 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:02.905158 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:37:02.909019 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:02.908987 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:37:02.911312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:02.911294 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:37:03.588002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:03.587969 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:04.084483 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:04.084442 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:06.794594 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:06.794559 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:07.103917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:07.103891 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:37:07.949233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:07.949197 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:08.221004 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:08.220910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:08.542549 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:08.542438 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:08.809740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:08.809660 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:09.136803 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.136764 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:09.386887 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.386796 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn"] Nov 04 03:37:09.387554 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.387529 4959 cpu_manager.go:414] "containerMap: removing container" podUID="05af6cd7-a401-4ed8-a220-3d4e45480d1c" containerName="revoke-keys" Nov 04 03:37:09.387554 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.387550 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05af6cd7-a401-4ed8-a220-3d4e45480d1c" containerName="revoke-keys" Nov 04 03:37:09.387818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.387738 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="05af6cd7-a401-4ed8-a220-3d4e45480d1c" containerName="revoke-keys" Nov 04 03:37:09.392728 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.392706 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.469657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.469568 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.469657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.469611 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.469893 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.469796 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcf2\" (UniqueName: \"kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.499202 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.499166 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:37:09.499453 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.499435 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:37:09.499592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.499492 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:37:09.499592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.499510 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:37:09.503986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.503965 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn"] Nov 04 03:37:09.553224 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.553192 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:09.571018 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.570981 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.571178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.571029 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.571178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.571123 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcf2\" (UniqueName: \"kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.571918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.571898 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.573614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.573594 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.632870 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.632839 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcf2\" (UniqueName: \"kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2\") pod \"maas-api-revoke-sub-eea5b7af-9lxnn\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.709816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.709728 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:09.801016 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:09.800989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:10.096693 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:10.096664 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn"] Nov 04 03:37:10.098139 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:37:10.098114 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd04779_e52d_4bf6_ad68_dfac94bdde16.slice/crio-1a85a41be313c9257427df2989403d3b14ca8288f15cc876ec7eacc9c213290f WatchSource:0}: Error finding container 1a85a41be313c9257427df2989403d3b14ca8288f15cc876ec7eacc9c213290f: Status 404 returned error can't find the container with id 1a85a41be313c9257427df2989403d3b14ca8288f15cc876ec7eacc9c213290f Nov 04 03:37:10.261876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:10.261842 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" event={"ID":"7bd04779-e52d-4bf6-ad68-dfac94bdde16","Type":"ContainerStarted","Data":"1a85a41be313c9257427df2989403d3b14ca8288f15cc876ec7eacc9c213290f"} Nov 04 03:37:10.885619 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:10.885584 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:11.046589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:11.046552 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:37:11.270059 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:11.269982 4959 generic.go:356] "Generic (PLEG): container finished" podID="7bd04779-e52d-4bf6-ad68-dfac94bdde16" containerID="472c72ed8e02b74ca63806acb1ee395557a223a596757dc6b6f7285d3b68cdba" exitCode=0 Nov 04 03:37:11.270422 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:11.270056 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" event={"ID":"7bd04779-e52d-4bf6-ad68-dfac94bdde16","Type":"ContainerDied","Data":"472c72ed8e02b74ca63806acb1ee395557a223a596757dc6b6f7285d3b68cdba"} Nov 04 03:37:12.451499 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.451457 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:12.523952 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.523924 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca\") pod \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " Nov 04 03:37:12.524095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.523994 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca\") pod \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " Nov 04 03:37:12.524095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.524015 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2\" (UniqueName: \"kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2\") pod \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\" (UID: \"7bd04779-e52d-4bf6-ad68-dfac94bdde16\") " Nov 04 03:37:12.524351 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.524324 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca" pod "7bd04779-e52d-4bf6-ad68-dfac94bdde16" (UID: "7bd04779-e52d-4bf6-ad68-dfac94bdde16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:37:12.526242 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.526221 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca" pod "7bd04779-e52d-4bf6-ad68-dfac94bdde16" (UID: "7bd04779-e52d-4bf6-ad68-dfac94bdde16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:37:12.526326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.526280 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2" pod "7bd04779-e52d-4bf6-ad68-dfac94bdde16" (UID: "7bd04779-e52d-4bf6-ad68-dfac94bdde16"). InnerVolumeSpecName "kube-api-access-5kcf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:37:12.625625 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.625585 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:12.625625 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.625617 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd04779-e52d-4bf6-ad68-dfac94bdde16-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:12.625625 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:12.625630 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kcf2\" (UniqueName: \"kubernetes.io/projected/7bd04779-e52d-4bf6-ad68-dfac94bdde16-kube-api-access-5kcf2\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:37:13.283622 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:13.283587 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" event={"ID":"7bd04779-e52d-4bf6-ad68-dfac94bdde16","Type":"ContainerDied","Data":"1a85a41be313c9257427df2989403d3b14ca8288f15cc876ec7eacc9c213290f"} Nov 04 03:37:13.283622 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:13.283622 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn" Nov 04 03:37:13.283841 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:13.283661 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a85a41be313c9257427df2989403d3b14ca8288f15cc876ec7eacc9c213290f" Nov 04 03:37:13.511889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:13.511851 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:14.221023 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:14.220988 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:16.354184 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:37:16.354151 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:37:16.606063 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.605968 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:16.656404 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.656364 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l"] Nov 04 03:37:16.657024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.657005 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7bd04779-e52d-4bf6-ad68-dfac94bdde16" containerName="revoke-keys" Nov 04 03:37:16.657024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.657022 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd04779-e52d-4bf6-ad68-dfac94bdde16" containerName="revoke-keys" Nov 04 03:37:16.657203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.657144 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd04779-e52d-4bf6-ad68-dfac94bdde16" containerName="revoke-keys" Nov 04 03:37:16.660659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.660642 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.765546 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.765515 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l"] Nov 04 03:37:16.776857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.776828 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ftj\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.776997 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.776960 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.776997 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.776986 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.777105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.777041 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.777105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.777085 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.777202 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.777106 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.777202 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.777180 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.777305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.777260 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.777350 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.777323 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878360 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878645 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878506 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878645 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878602 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ftj\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878758 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878736 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878773 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878797 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878812 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878868 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.878918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878895 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.879082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.878942 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.879260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.879237 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.879738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.879684 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.879738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.879567 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.880588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.880567 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.881903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.881882 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.882767 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.882738 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.943052 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.943024 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ftj\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.943716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.943690 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token\") pod \"e2e-extmod-b-491fe7-openshift-default-7774566777-6957l\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:16.974043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:16.973838 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:17.439170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:17.439133 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l"] Nov 04 03:37:17.497365 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:17.497319 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:37:17.497649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:17.497624 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:37:17.497807 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:17.497789 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:37:17.517519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:17.517049 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:17.764637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:17.764534 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:18.314910 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:18.314876 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" event={"ID":"e41fd06a-2bc4-4951-8023-a5436e88ca03","Type":"ContainerStarted","Data":"873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59"} Nov 04 03:37:18.315072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:18.314939 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" event={"ID":"e41fd06a-2bc4-4951-8023-a5436e88ca03","Type":"ContainerStarted","Data":"64c137ab0a05cee3a54695692e27a931d447052f07b8948f41ae1f4cf45cc38f"} Nov 04 03:37:18.495505 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:18.488883 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podStartSLOduration=2.488862662 podStartE2EDuration="2.488862662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:37:18.486261932 +0000 UTC m=+1215.869689975" watchObservedRunningTime="2026-11-04 03:37:18.488862662 +0000 UTC m=+1215.872290702" Nov 04 03:37:18.981835 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:18.981802 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:18.993639 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:18.993603 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 03:37:18.994697 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:18.994664 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 03:37:19.029459 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:19.029420 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:19.974887 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:19.974858 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 03:37:19.975284 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:19.974915 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 03:37:20.974267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:20.974231 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 03:37:20.974443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:20.974282 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 03:37:21.046329 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:21.046293 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:37:21.974573 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:21.974534 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 03:37:21.974783 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:21.974595 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 03:37:22.853083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:22.853050 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:22.974435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:22.974403 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 04 03:37:22.974592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:22.974450 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 04 03:37:23.837142 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:23.837085 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:23.979270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:23.979239 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:23.979771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:23.979629 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:23.980327 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:23.980310 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:37:28.733822 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:28.733784 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:30.335274 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:30.335236 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:30.640081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:30.639983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:31.047000 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:31.046922 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:37:31.588092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:31.588059 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:33.791335 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:33.791307 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:38.349752 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:38.349714 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:38.649993 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:38.649909 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:38.965271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:38.965180 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:41.047161 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:41.047125 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:37:43.482456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:43.482012 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:43.732712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:43.732611 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:46.354860 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:37:46.354823 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:37:48.542863 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:48.542828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:51.047203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:51.047166 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:37:52.853516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:52.853457 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:53.347581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:53.347545 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:54.876986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:54.876944 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:37:58.881148 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:58.881110 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:37:59.798398 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:37:59.798346 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:01.046903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:01.046853 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:03.332019 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:03.331983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:05.174396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:05.174359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:05.805544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:05.805511 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp"] Nov 04 03:38:05.805750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:05.805702 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" containerID="cri-o://b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861" gracePeriod=30 Nov 04 03:38:06.974241 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:06.974206 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:07.442436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:07.442391 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:08.548255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:08.548222 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:08.956355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:08.956315 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst"] Nov 04 03:38:08.956577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:08.956545 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" containerID="cri-o://1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9" gracePeriod=30 Nov 04 03:38:09.091642 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:09.091547 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:10.271275 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:10.271234 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:10.647907 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:10.647873 4959 patch_prober.go:29] interesting pod/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:38:10.648077 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:10.647936 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:38:11.046389 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.046353 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:11.061750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.061727 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:38:11.145170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145131 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145177 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145223 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145285 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145327 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145352 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145368 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145402 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145505 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:11.145796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145544 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket\") pod \"71a6af5b-4d34-4822-819b-1e40e917a36c\" (UID: \"71a6af5b-4d34-4822-819b-1e40e917a36c\") " Nov 04 03:38:11.145796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145743 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:11.145944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.145805 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:11.146065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146041 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:11.146262 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146104 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:11.146390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146324 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.146390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146346 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a6af5b-4d34-4822-819b-1e40e917a36c-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.146390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146361 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.146390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146376 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.146390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.146389 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.148393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.148011 4959 patch_prober.go:29] interesting pod/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:38:11.148393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.148189 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:38:11.148393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.148211 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:38:11.148393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.148243 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "kube-api-access-hrngh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:11.148738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.148717 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:11.148823 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.148763 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token" pod "71a6af5b-4d34-4822-819b-1e40e917a36c" (UID: "71a6af5b-4d34-4822-819b-1e40e917a36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:11.247763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.247686 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrngh\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-kube-api-access-hrngh\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.247763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.247713 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.247763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.247724 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.247763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.247732 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a6af5b-4d34-4822-819b-1e40e917a36c-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:11.711136 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.711098 4959 generic.go:356] "Generic (PLEG): container finished" podID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerID="b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861" exitCode=0 Nov 04 03:38:11.711665 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.711149 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" Nov 04 03:38:11.711665 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.711195 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" event={"ID":"71a6af5b-4d34-4822-819b-1e40e917a36c","Type":"ContainerDied","Data":"b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861"} Nov 04 03:38:11.711665 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.711263 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp" event={"ID":"71a6af5b-4d34-4822-819b-1e40e917a36c","Type":"ContainerDied","Data":"c5e63337ff1cc571173f439602a684d48d7acb94a206d1299c867a73ac55dca9"} Nov 04 03:38:11.711665 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.711289 4959 scope.go:122] "RemoveContainer" containerID="b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861" Nov 04 03:38:11.728045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.728024 4959 scope.go:122] "RemoveContainer" containerID="b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861" Nov 04 03:38:11.728343 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:38:11.728315 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861\": container with ID starting with b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861 not found: ID does not exist" containerID="b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861" Nov 04 03:38:11.728393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.728356 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861"} err="failed to get container status \"b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861\": rpc error: code = NotFound desc = could not find container \"b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861\": container with ID starting with b60200bf313e8fbb6359e4efb61a1d0c6e56d46f228f6f9f92b4f5d924e51861 not found: ID does not exist" Nov 04 03:38:11.882845 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.882808 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp"] Nov 04 03:38:11.932533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:11.932498 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-90051a-openshift-default-5d4df756c5-gj4fp"] Nov 04 03:38:12.800812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:12.800782 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" path="/var/lib/kubelet/pods/71a6af5b-4d34-4822-819b-1e40e917a36c/volumes" Nov 04 03:38:12.802347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:12.802322 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:14.224294 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.224242 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.393894 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.393958 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394035 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394116 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394144 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394198 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394233 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394287 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.394682 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.395104 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.395372 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.395484 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.395553 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket\") pod \"c87f2944-7006-42a2-b99e-34a92cf65610\" (UID: \"c87f2944-7006-42a2-b99e-34a92cf65610\") " Nov 04 03:38:14.397511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.396003 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.399928 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.399904 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:38:14.400750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.400686 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:38:14.400876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.400796 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "kube-api-access-shnwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:14.400876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.400846 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token" pod "c87f2944-7006-42a2-b99e-34a92cf65610" (UID: "c87f2944-7006-42a2-b99e-34a92cf65610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:14.404309 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404286 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404311 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c87f2944-7006-42a2-b99e-34a92cf65610-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404324 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c87f2944-7006-42a2-b99e-34a92cf65610-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404336 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404347 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404359 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404372 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404385 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shnwf\" (UniqueName: \"kubernetes.io/projected/c87f2944-7006-42a2-b99e-34a92cf65610-kube-api-access-shnwf\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.404433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.404411 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c87f2944-7006-42a2-b99e-34a92cf65610-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:14.731732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.731700 4959 generic.go:356] "Generic (PLEG): container finished" podID="c87f2944-7006-42a2-b99e-34a92cf65610" containerID="1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9" exitCode=0 Nov 04 03:38:14.731917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.731752 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" Nov 04 03:38:14.731917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.731775 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" event={"ID":"c87f2944-7006-42a2-b99e-34a92cf65610","Type":"ContainerDied","Data":"1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9"} Nov 04 03:38:14.731917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.731815 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst" event={"ID":"c87f2944-7006-42a2-b99e-34a92cf65610","Type":"ContainerDied","Data":"cb2fca2a7ceb8930d7007243d1fd38ce2dee84f27a637ac04fd68a5c5dd4517c"} Nov 04 03:38:14.731917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.731836 4959 scope.go:122] "RemoveContainer" containerID="1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9" Nov 04 03:38:14.742635 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.742616 4959 scope.go:122] "RemoveContainer" containerID="4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83" Nov 04 03:38:14.751480 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.751448 4959 scope.go:122] "RemoveContainer" containerID="1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9" Nov 04 03:38:14.751776 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:38:14.751753 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9\": container with ID starting with 1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9 not found: ID does not exist" containerID="1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9" Nov 04 03:38:14.751851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.751787 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9"} err="failed to get container status \"1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9\": rpc error: code = NotFound desc = could not find container \"1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9\": container with ID starting with 1e6da7d85becde1ab2a82ef7328294577d37af40ed65a39410eb749fc060a2c9 not found: ID does not exist" Nov 04 03:38:14.751851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.751815 4959 scope.go:122] "RemoveContainer" containerID="4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83" Nov 04 03:38:14.752051 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:38:14.752033 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83\": container with ID starting with 4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83 not found: ID does not exist" containerID="4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83" Nov 04 03:38:14.752103 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.752057 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83"} err="failed to get container status \"4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83\": rpc error: code = NotFound desc = could not find container \"4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83\": container with ID starting with 4e48a0494e33c370bf403b50ec8bfb175995c005938062033b0dbc70690ddd83 not found: ID does not exist" Nov 04 03:38:14.900443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.900407 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst"] Nov 04 03:38:14.951109 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:14.951073 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36ccd3-openshift-default-f57d8465-l4fst"] Nov 04 03:38:15.330848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:15.330808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:16.355405 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:38:16.355372 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:38:16.799665 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:16.799576 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" path="/var/lib/kubelet/pods/c87f2944-7006-42a2-b99e-34a92cf65610/volumes" Nov 04 03:38:20.140659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:20.140611 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:22.803065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:22.803029 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:25.019887 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:25.019853 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:29.919719 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:29.919679 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:32.803370 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:32.803332 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:34.756018 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:34.755985 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:34.800852 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:34.800819 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:38:38.626131 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626091 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q"] Nov 04 03:38:38.626757 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626738 4959 cpu_manager.go:414] "containerMap: removing container" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" Nov 04 03:38:38.626814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626759 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" Nov 04 03:38:38.626814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626789 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" Nov 04 03:38:38.626814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626794 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" Nov 04 03:38:38.626910 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626886 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" Nov 04 03:38:38.626910 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.626898 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a6af5b-4d34-4822-819b-1e40e917a36c" containerName="istio-proxy" Nov 04 03:38:38.630404 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.630387 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.687991 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.687963 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q"] Nov 04 03:38:38.734815 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.734783 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:38:38.734815 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.734785 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:38:38.735048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.734965 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:38:38.735048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.734985 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:38:38.735423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.735400 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzq6\" (UniqueName: \"kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.735542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.735452 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.735633 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.735621 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.836655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.836624 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.836849 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.836686 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzq6\" (UniqueName: \"kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.836849 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.836721 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.837339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.837319 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.839125 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.839102 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.898617 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.898543 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzq6\" (UniqueName: \"kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6\") pod \"maas-api-revoke-sub-b3884c08-dbv9q\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:38.946258 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:38.946229 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:39.320880 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:39.320843 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q"] Nov 04 03:38:39.321952 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:38:39.321926 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83efd1f1_28e0_4b79_862e_8ec83bfe3e8b.slice/crio-21e987314e3d09b35d1d9c5d3b1857dbe3be7009ae5729c168ab005eae2a63bc WatchSource:0}: Error finding container 21e987314e3d09b35d1d9c5d3b1857dbe3be7009ae5729c168ab005eae2a63bc: Status 404 returned error can't find the container with id 21e987314e3d09b35d1d9c5d3b1857dbe3be7009ae5729c168ab005eae2a63bc Nov 04 03:38:39.627354 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:39.627319 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:39.883553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:39.883438 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" event={"ID":"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b","Type":"ContainerStarted","Data":"c8e885a9dc372c93de5315d880cd926a0c358093cae6fc2a3b4f1624e3c0c812"} Nov 04 03:38:39.883770 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:39.883577 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" event={"ID":"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b","Type":"ContainerStarted","Data":"21e987314e3d09b35d1d9c5d3b1857dbe3be7009ae5729c168ab005eae2a63bc"} Nov 04 03:38:40.002455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:40.002404 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" podStartSLOduration=2.002391444 podStartE2EDuration="2.002391444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:38:40.000738778 +0000 UTC m=+1297.384166854" watchObservedRunningTime="2026-11-04 03:38:40.002391444 +0000 UTC m=+1297.385819484" Nov 04 03:38:40.891146 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:40.891107 4959 generic.go:356] "Generic (PLEG): container finished" podID="83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" containerID="c8e885a9dc372c93de5315d880cd926a0c358093cae6fc2a3b4f1624e3c0c812" exitCode=0 Nov 04 03:38:40.891601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:40.891191 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" event={"ID":"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b","Type":"ContainerDied","Data":"c8e885a9dc372c93de5315d880cd926a0c358093cae6fc2a3b4f1624e3c0c812"} Nov 04 03:38:42.063291 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.063264 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:42.175394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.175307 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6\" (UniqueName: \"kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6\") pod \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " Nov 04 03:38:42.175598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.175406 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca\") pod \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " Nov 04 03:38:42.175598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.175445 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca\") pod \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\" (UID: \"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b\") " Nov 04 03:38:42.175857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.175831 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca" pod "83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" (UID: "83efd1f1-28e0-4b79-862e-8ec83bfe3e8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:42.176022 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.176006 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:42.177785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.177764 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6" pod "83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" (UID: "83efd1f1-28e0-4b79-862e-8ec83bfe3e8b"). InnerVolumeSpecName "kube-api-access-6kzq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:42.178032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.178010 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca" pod "83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" (UID: "83efd1f1-28e0-4b79-862e-8ec83bfe3e8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:38:42.277603 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.277568 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzq6\" (UniqueName: \"kubernetes.io/projected/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-kube-api-access-6kzq6\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:42.277603 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.277600 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:42.802612 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.802577 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:42.904116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.904084 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" Nov 04 03:38:42.904116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.904094 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q" event={"ID":"83efd1f1-28e0-4b79-862e-8ec83bfe3e8b","Type":"ContainerDied","Data":"21e987314e3d09b35d1d9c5d3b1857dbe3be7009ae5729c168ab005eae2a63bc"} Nov 04 03:38:42.904379 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:42.904143 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21e987314e3d09b35d1d9c5d3b1857dbe3be7009ae5729c168ab005eae2a63bc" Nov 04 03:38:44.552492 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:44.552443 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:46.356873 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:38:46.356832 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:38:48.506282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506245 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl"] Nov 04 03:38:48.506825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506811 4959 cpu_manager.go:414] "containerMap: removing container" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" Nov 04 03:38:48.506869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506827 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" Nov 04 03:38:48.506869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506867 4959 cpu_manager.go:414] "containerMap: removing container" podUID="83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" containerName="revoke-keys" Nov 04 03:38:48.506952 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506873 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" containerName="revoke-keys" Nov 04 03:38:48.506982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506959 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" containerName="revoke-keys" Nov 04 03:38:48.506982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.506971 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87f2944-7006-42a2-b99e-34a92cf65610" containerName="istio-proxy" Nov 04 03:38:48.510253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.510236 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.548112 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.548077 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl"] Nov 04 03:38:48.563733 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.563700 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:38:48.563967 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.563740 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:38:48.604007 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.603975 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:38:48.604359 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.604342 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:38:48.608055 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.607776 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.608055 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.607839 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt5g5\" (UniqueName: \"kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.608055 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.607914 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.709536 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.709464 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.709795 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.709574 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt5g5\" (UniqueName: \"kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.709795 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.709649 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.710364 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.710341 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.712321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.712300 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.771913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.771840 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt5g5\" (UniqueName: \"kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5\") pod \"maas-api-revoke-sub-f3529b6f-jplfl\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:48.824013 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:48.823976 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:49.211399 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:49.211371 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl"] Nov 04 03:38:49.212856 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:38:49.212827 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b5cc2e_25df_4e48_a334_987d9aa3cd03.slice/crio-462b5ae4b99efb229d965d8dbf34b746d9a14222a879ed5231863e26401be144 WatchSource:0}: Error finding container 462b5ae4b99efb229d965d8dbf34b746d9a14222a879ed5231863e26401be144: Status 404 returned error can't find the container with id 462b5ae4b99efb229d965d8dbf34b746d9a14222a879ed5231863e26401be144 Nov 04 03:38:49.575772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:49.575736 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:49.953672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:49.953633 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" event={"ID":"07b5cc2e-25df-4e48-a334-987d9aa3cd03","Type":"ContainerStarted","Data":"f1cc4619c603d393039cfa0daed99234f9bb6e09882fe8f0eae71f93e19ab333"} Nov 04 03:38:49.953866 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:49.953716 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" event={"ID":"07b5cc2e-25df-4e48-a334-987d9aa3cd03","Type":"ContainerStarted","Data":"462b5ae4b99efb229d965d8dbf34b746d9a14222a879ed5231863e26401be144"} Nov 04 03:38:50.071432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:50.071368 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" podStartSLOduration=2.071353924 podStartE2EDuration="2.071353924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:38:50.070543638 +0000 UTC m=+1307.453971669" watchObservedRunningTime="2026-11-04 03:38:50.071353924 +0000 UTC m=+1307.454782019" Nov 04 03:38:50.189740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:50.189703 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:50.963335 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:50.963300 4959 generic.go:356] "Generic (PLEG): container finished" podID="07b5cc2e-25df-4e48-a334-987d9aa3cd03" containerID="f1cc4619c603d393039cfa0daed99234f9bb6e09882fe8f0eae71f93e19ab333" exitCode=0 Nov 04 03:38:50.963887 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:50.963390 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" event={"ID":"07b5cc2e-25df-4e48-a334-987d9aa3cd03","Type":"ContainerDied","Data":"f1cc4619c603d393039cfa0daed99234f9bb6e09882fe8f0eae71f93e19ab333"} Nov 04 03:38:51.385149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:51.385116 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:51.675285 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:51.675201 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:52.010239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.010095 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:52.184822 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.184789 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:52.276544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.260939 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5\" (UniqueName: \"kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5\") pod \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " Nov 04 03:38:52.276544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.260994 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca\") pod \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " Nov 04 03:38:52.276544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.261074 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca\") pod \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\" (UID: \"07b5cc2e-25df-4e48-a334-987d9aa3cd03\") " Nov 04 03:38:52.276544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.263858 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca" pod "07b5cc2e-25df-4e48-a334-987d9aa3cd03" (UID: "07b5cc2e-25df-4e48-a334-987d9aa3cd03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:38:52.276544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.269241 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5" pod "07b5cc2e-25df-4e48-a334-987d9aa3cd03" (UID: "07b5cc2e-25df-4e48-a334-987d9aa3cd03"). InnerVolumeSpecName "kube-api-access-pt5g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:38:52.276544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.271357 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca" pod "07b5cc2e-25df-4e48-a334-987d9aa3cd03" (UID: "07b5cc2e-25df-4e48-a334-987d9aa3cd03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:38:52.292454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.292394 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:52.302646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.301732 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:52.302646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.301761 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt5g5\" (UniqueName: \"kubernetes.io/projected/07b5cc2e-25df-4e48-a334-987d9aa3cd03-kube-api-access-pt5g5\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:52.302646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.301782 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b5cc2e-25df-4e48-a334-987d9aa3cd03-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:38:52.566705 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.566608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:52.802847 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.802809 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:38:52.988620 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.988593 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" Nov 04 03:38:52.988780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.988592 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl" event={"ID":"07b5cc2e-25df-4e48-a334-987d9aa3cd03","Type":"ContainerDied","Data":"462b5ae4b99efb229d965d8dbf34b746d9a14222a879ed5231863e26401be144"} Nov 04 03:38:52.988780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:52.988723 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="462b5ae4b99efb229d965d8dbf34b746d9a14222a879ed5231863e26401be144" Nov 04 03:38:53.118240 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.118200 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:53.380857 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.380785 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:53.996253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.996220 4959 generic.go:356] "Generic (PLEG): container finished" podID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerID="6b9e28695899ba0892316b1aba344e5b3e429e0f1bd26ae5d6cb6cc6ec282fb6" exitCode=0 Nov 04 03:38:53.996432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.996274 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" event={"ID":"6df7a391-1ace-4782-8f2c-410d14c4429e","Type":"ContainerDied","Data":"6b9e28695899ba0892316b1aba344e5b3e429e0f1bd26ae5d6cb6cc6ec282fb6"} Nov 04 03:38:53.996676 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.996662 4959 scope.go:122] "RemoveContainer" containerID="6b9e28695899ba0892316b1aba344e5b3e429e0f1bd26ae5d6cb6cc6ec282fb6" Nov 04 03:38:53.998193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.998140 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:38:53.998263 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.998234 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:38:53.998302 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:53.998266 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:38:54.570275 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:54.570110 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:55.012042 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:55.012002 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" event={"ID":"6df7a391-1ace-4782-8f2c-410d14c4429e","Type":"ContainerStarted","Data":"b709a3054f84ced36c6720f074147e22213d6750d3d761dfd6b94d8bfa1e8890"} Nov 04 03:38:55.569066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:55.569023 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:55.611149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:55.610992 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:38:55.611149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:55.611133 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 03:38:55.611740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:55.611177 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 03:38:56.606518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.605380 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5"] Nov 04 03:38:56.606518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.605964 4959 cpu_manager.go:414] "containerMap: removing container" podUID="07b5cc2e-25df-4e48-a334-987d9aa3cd03" containerName="revoke-keys" Nov 04 03:38:56.606518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.605977 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b5cc2e-25df-4e48-a334-987d9aa3cd03" containerName="revoke-keys" Nov 04 03:38:56.606518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.606097 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b5cc2e-25df-4e48-a334-987d9aa3cd03" containerName="revoke-keys" Nov 04 03:38:56.609779 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.609761 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.610282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.610171 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 03:38:56.610282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.610224 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 03:38:56.666968 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.666934 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5"] Nov 04 03:38:56.712566 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.712530 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5d459a\"/\"kube-root-ca.crt\"" Nov 04 03:38:56.712800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.712594 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5d459a\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5d459a\"" Nov 04 03:38:56.712800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.712632 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5d459a\"/\"openshift-service-ca.crt\"" Nov 04 03:38:56.712800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.712667 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5d459a\"/\"agc-cred-49692ad7a38d\"" Nov 04 03:38:56.712800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.712690 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5d459a\"/\"payload-processing-external-model-e2e--7b78efea-dockercfg-952g2\"" Nov 04 03:38:56.714209 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.714191 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5d459a\"/\"routing-overlay\"" Nov 04 03:38:56.716171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.716146 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.716307 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.716217 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.716391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.716367 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jznfr\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.716458 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.716438 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.817884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.817854 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.818090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.817934 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.818090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.817989 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jznfr\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.818090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.818019 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.818681 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.818655 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.819095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.819077 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.820820 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.820799 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.906238 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.906167 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jznfr\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr\") pod \"payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:56.923879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:56.923844 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:57.173400 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:57.172441 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:57.345108 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:57.345076 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5"] Nov 04 03:38:57.347036 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:38:57.346990 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode636a527_f551_4194_b91c_af45e9237e56.slice/crio-d262fbc83c16bf695c3106fa3a68bb25e76888de60080fd84520ac1300380ca2 WatchSource:0}: Error finding container d262fbc83c16bf695c3106fa3a68bb25e76888de60080fd84520ac1300380ca2: Status 404 returned error can't find the container with id d262fbc83c16bf695c3106fa3a68bb25e76888de60080fd84520ac1300380ca2 Nov 04 03:38:57.611950 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:57.611910 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 03:38:57.612204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:57.612174 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 03:38:58.034865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:58.034825 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" event={"ID":"e636a527-f551-4194-b91c-af45e9237e56","Type":"ContainerStarted","Data":"1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c"} Nov 04 03:38:58.035253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:58.034882 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" event={"ID":"e636a527-f551-4194-b91c-af45e9237e56","Type":"ContainerStarted","Data":"d262fbc83c16bf695c3106fa3a68bb25e76888de60080fd84520ac1300380ca2"} Nov 04 03:38:58.035418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:58.035393 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:38:58.164915 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:58.164844 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" podStartSLOduration=2.164823481 podStartE2EDuration="2.164823481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:38:58.159674839 +0000 UTC m=+1315.543102865" watchObservedRunningTime="2026-11-04 03:38:58.164823481 +0000 UTC m=+1315.548251525" Nov 04 03:38:58.610144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:58.610104 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 03:38:58.610330 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:58.610164 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 03:38:59.311496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:59.311102 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:59.569257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:59.569161 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:38:59.610278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:59.610230 4959 patch_prober.go:29] interesting pod/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 04 03:38:59.610498 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:38:59.610296 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 04 03:39:00.139025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:00.138992 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:00.614998 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:00.614970 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:39:00.615501 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:00.615390 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:39:00.616149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:00.616130 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:39:02.567500 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:02.567452 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:39:02.814179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:02.814134 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:04.246987 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:04.246955 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:05.912511 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:05.912440 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:06.232131 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:06.232042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:06.566516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:06.566402 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:09.043903 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:09.043876 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:39:09.137949 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:09.137915 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:11.117233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:11.117195 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:11.420745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:11.420649 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:11.459394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:11.459363 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v"] Nov 04 03:39:11.721858 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:11.721760 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:11.976284 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:11.976191 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:12.544837 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:12.544798 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:12.803367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:12.803258 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:13.071171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:13.071078 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:14.058698 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:14.058658 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:14.106292 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:14.106256 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j"] Nov 04 03:39:14.215088 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:14.215054 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m"] Nov 04 03:39:14.643598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:14.643549 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:14.956117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:14.956027 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:16.354527 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:39:16.354492 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:39:16.460592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.460551 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" podUID="2950b8be-019d-4e7f-974c-ca4b9d64a404" containerName="payload-processing" containerID="cri-o://9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697" gracePeriod=25 Nov 04 03:39:16.730604 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.730579 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:39:16.817550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.817520 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume\") pod \"2950b8be-019d-4e7f-974c-ca4b9d64a404\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " Nov 04 03:39:16.817746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.817594 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n\") pod \"2950b8be-019d-4e7f-974c-ca4b9d64a404\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " Nov 04 03:39:16.817746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.817615 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume\") pod \"2950b8be-019d-4e7f-974c-ca4b9d64a404\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " Nov 04 03:39:16.817746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.817678 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials\") pod \"2950b8be-019d-4e7f-974c-ca4b9d64a404\" (UID: \"2950b8be-019d-4e7f-974c-ca4b9d64a404\") " Nov 04 03:39:16.817974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.817950 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume" pod "2950b8be-019d-4e7f-974c-ca4b9d64a404" (UID: "2950b8be-019d-4e7f-974c-ca4b9d64a404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:16.818037 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.817982 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume" pod "2950b8be-019d-4e7f-974c-ca4b9d64a404" (UID: "2950b8be-019d-4e7f-974c-ca4b9d64a404"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:16.818134 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.818120 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:16.818171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.818138 4959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2950b8be-019d-4e7f-974c-ca4b9d64a404-routing-overlay-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:16.820142 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.820120 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n" pod "2950b8be-019d-4e7f-974c-ca4b9d64a404" (UID: "2950b8be-019d-4e7f-974c-ca4b9d64a404"). InnerVolumeSpecName "kube-api-access-jv47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:16.820215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.820195 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials" pod "2950b8be-019d-4e7f-974c-ca4b9d64a404" (UID: "2950b8be-019d-4e7f-974c-ca4b9d64a404"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:16.919925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.919842 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv47n\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-kube-api-access-jv47n\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:16.919925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:16.919878 4959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2950b8be-019d-4e7f-974c-ca4b9d64a404-provider-credentials\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:17.171378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.171285 4959 generic.go:356] "Generic (PLEG): container finished" podID="2950b8be-019d-4e7f-974c-ca4b9d64a404" containerID="9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697" exitCode=0 Nov 04 03:39:17.171378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.171333 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" Nov 04 03:39:17.171643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.171379 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" event={"ID":"2950b8be-019d-4e7f-974c-ca4b9d64a404","Type":"ContainerDied","Data":"9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697"} Nov 04 03:39:17.171643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.171442 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v" event={"ID":"2950b8be-019d-4e7f-974c-ca4b9d64a404","Type":"ContainerDied","Data":"7d931b822f536a11a820349ad7a13fd2a24f8619afa0f98ab5621a07973a4755"} Nov 04 03:39:17.171643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.171488 4959 scope.go:122] "RemoveContainer" containerID="9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697" Nov 04 03:39:17.182684 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.182665 4959 scope.go:122] "RemoveContainer" containerID="9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697" Nov 04 03:39:17.182937 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:39:17.182915 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697\": container with ID starting with 9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697 not found: ID does not exist" containerID="9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697" Nov 04 03:39:17.183032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.182939 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697"} err="failed to get container status \"9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697\": rpc error: code = NotFound desc = could not find container \"9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697\": container with ID starting with 9d9b07d23630ef9ea7cd0822e220ed62f75ce88e0c57d9146bf016501b9b1697 not found: ID does not exist" Nov 04 03:39:17.341328 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.341294 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v"] Nov 04 03:39:17.390600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:17.390570 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d1e2ab/payload-processing-external-model-e2e-extmod-b-d1e2ab-844c6bz4v"] Nov 04 03:39:18.562681 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:18.562650 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:18.799948 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:18.799915 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2950b8be-019d-4e7f-974c-ca4b9d64a404" path="/var/lib/kubelet/pods/2950b8be-019d-4e7f-974c-ca4b9d64a404/volumes" Nov 04 03:39:18.862785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:18.862752 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:19.109494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.108715 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" podUID="b658291f-ee09-429e-8b9f-5ddf99384ed3" containerName="payload-processing" containerID="cri-o://7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60" gracePeriod=25 Nov 04 03:39:19.131141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.131108 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:19.216024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.215988 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" podUID="a394455d-d89b-4513-b82b-22db02ea07ea" containerName="payload-pre-processing" containerID="cri-o://fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38" gracePeriod=25 Nov 04 03:39:19.386545 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.386505 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:19.422643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.422615 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:39:19.474226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.474204 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:39:19.554927 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.554898 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728\" (UniqueName: \"kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728\") pod \"a394455d-d89b-4513-b82b-22db02ea07ea\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " Nov 04 03:39:19.555072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.554962 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume\") pod \"a394455d-d89b-4513-b82b-22db02ea07ea\" (UID: \"a394455d-d89b-4513-b82b-22db02ea07ea\") " Nov 04 03:39:19.555072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.555057 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6\" (UniqueName: \"kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6\") pod \"b658291f-ee09-429e-8b9f-5ddf99384ed3\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " Nov 04 03:39:19.555175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.555105 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume\") pod \"b658291f-ee09-429e-8b9f-5ddf99384ed3\" (UID: \"b658291f-ee09-429e-8b9f-5ddf99384ed3\") " Nov 04 03:39:19.555536 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.555507 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume" pod "a394455d-d89b-4513-b82b-22db02ea07ea" (UID: "a394455d-d89b-4513-b82b-22db02ea07ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:19.555666 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.555532 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume" pod "b658291f-ee09-429e-8b9f-5ddf99384ed3" (UID: "b658291f-ee09-429e-8b9f-5ddf99384ed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:19.555766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.555741 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a394455d-d89b-4513-b82b-22db02ea07ea-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:19.555818 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.555772 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b658291f-ee09-429e-8b9f-5ddf99384ed3-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:19.557952 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.557925 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728" pod "a394455d-d89b-4513-b82b-22db02ea07ea" (UID: "a394455d-d89b-4513-b82b-22db02ea07ea"). InnerVolumeSpecName "kube-api-access-sq728". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:19.558039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.557967 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6" pod "b658291f-ee09-429e-8b9f-5ddf99384ed3" (UID: "b658291f-ee09-429e-8b9f-5ddf99384ed3"). InnerVolumeSpecName "kube-api-access-nrsw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:19.656876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.656801 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq728\" (UniqueName: \"kubernetes.io/projected/a394455d-d89b-4513-b82b-22db02ea07ea-kube-api-access-sq728\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:19.656876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:19.656830 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrsw6\" (UniqueName: \"kubernetes.io/projected/b658291f-ee09-429e-8b9f-5ddf99384ed3-kube-api-access-nrsw6\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:20.202485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.202421 4959 generic.go:356] "Generic (PLEG): container finished" podID="b658291f-ee09-429e-8b9f-5ddf99384ed3" containerID="7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60" exitCode=0 Nov 04 03:39:20.202670 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.202587 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" Nov 04 03:39:20.202670 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.202593 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" event={"ID":"b658291f-ee09-429e-8b9f-5ddf99384ed3","Type":"ContainerDied","Data":"7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60"} Nov 04 03:39:20.202780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.202706 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j" event={"ID":"b658291f-ee09-429e-8b9f-5ddf99384ed3","Type":"ContainerDied","Data":"2d9ddf7573516f3b096dc663a9c86d459ea6b72fd283f935aa030598cc344fef"} Nov 04 03:39:20.202780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.202737 4959 scope.go:122] "RemoveContainer" containerID="7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60" Nov 04 03:39:20.206690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.206573 4959 generic.go:356] "Generic (PLEG): container finished" podID="a394455d-d89b-4513-b82b-22db02ea07ea" containerID="fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38" exitCode=0 Nov 04 03:39:20.206690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.206620 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" Nov 04 03:39:20.206690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.206670 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" event={"ID":"a394455d-d89b-4513-b82b-22db02ea07ea","Type":"ContainerDied","Data":"fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38"} Nov 04 03:39:20.206872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.206706 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m" event={"ID":"a394455d-d89b-4513-b82b-22db02ea07ea","Type":"ContainerDied","Data":"42240eb6ecc5ba9493e2276d4a8eb35ec4d7bd4bee7835ae7f8e5935cd063dc6"} Nov 04 03:39:20.219031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.219012 4959 scope.go:122] "RemoveContainer" containerID="7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60" Nov 04 03:39:20.219613 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:39:20.219578 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60\": container with ID starting with 7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60 not found: ID does not exist" containerID="7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60" Nov 04 03:39:20.219730 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.219618 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60"} err="failed to get container status \"7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60\": rpc error: code = NotFound desc = could not find container \"7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60\": container with ID starting with 7b7979cde09fb9677e6ed3a2ebe352dbca1768c6f16b785913151f298026da60 not found: ID does not exist" Nov 04 03:39:20.219730 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.219636 4959 scope.go:122] "RemoveContainer" containerID="fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38" Nov 04 03:39:20.230331 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.230311 4959 scope.go:122] "RemoveContainer" containerID="fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38" Nov 04 03:39:20.230689 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:39:20.230666 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38\": container with ID starting with fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38 not found: ID does not exist" containerID="fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38" Nov 04 03:39:20.230778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.230696 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38"} err="failed to get container status \"fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38\": rpc error: code = NotFound desc = could not find container \"fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38\": container with ID starting with fd80bfc17d8a9c8c2b2534bbc61fcadce6fd809747acc476a0e83a3a9058ca38 not found: ID does not exist" Nov 04 03:39:20.369593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.369565 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m"] Nov 04 03:39:20.419940 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.419865 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-141891-77cfdd8587-hbt8m"] Nov 04 03:39:20.483658 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.483628 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j"] Nov 04 03:39:20.490118 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.490093 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-141891-699b689f5b-nmk6j"] Nov 04 03:39:20.800557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.800459 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a394455d-d89b-4513-b82b-22db02ea07ea" path="/var/lib/kubelet/pods/a394455d-d89b-4513-b82b-22db02ea07ea/volumes" Nov 04 03:39:20.800917 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:20.800866 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b658291f-ee09-429e-8b9f-5ddf99384ed3" path="/var/lib/kubelet/pods/b658291f-ee09-429e-8b9f-5ddf99384ed3/volumes" Nov 04 03:39:22.802615 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:22.802574 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:23.937488 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:23.937446 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:25.109197 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:25.109167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:25.411144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:25.411050 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:25.684431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:25.684349 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:28.730560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:28.730504 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:31.102152 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:31.102114 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q"] Nov 04 03:39:31.214656 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:31.214620 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml"] Nov 04 03:39:31.576548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:31.576348 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:31.834464 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:31.834328 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:32.547525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:32.542910 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:32.803346 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:32.803261 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:33.770402 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:33.770360 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:34.102076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:34.102048 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:39:36.102809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:36.102767 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerName="payload-processing" containerID="cri-o://5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8" gracePeriod=25 Nov 04 03:39:36.218036 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:36.217987 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerName="payload-pre-processing" containerID="cri-o://5970cd707b4dc34385c075e5d60025f2c72d6e09b413833e70ea5a33c36f8b69" gracePeriod=25 Nov 04 03:39:37.509886 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:37.509847 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:38.294960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:38.294919 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:38.515210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:38.515168 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:38.839109 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:38.839058 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:39.792694 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:39.792659 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:43.769690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:43.769652 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:45.257445 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:45.257394 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:46.336074 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:46.336039 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:46.471500 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:39:46.471428 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:39:47.509394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:47.509354 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:48.514784 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:48.514751 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:48.626484 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:48.626433 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:49.732850 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.732811 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk"] Nov 04 03:39:49.733561 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733542 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2950b8be-019d-4e7f-974c-ca4b9d64a404" containerName="payload-processing" Nov 04 03:39:49.733674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733564 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2950b8be-019d-4e7f-974c-ca4b9d64a404" containerName="payload-processing" Nov 04 03:39:49.733674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733609 4959 cpu_manager.go:414] "containerMap: removing container" podUID="b658291f-ee09-429e-8b9f-5ddf99384ed3" containerName="payload-processing" Nov 04 03:39:49.733674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733617 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b658291f-ee09-429e-8b9f-5ddf99384ed3" containerName="payload-processing" Nov 04 03:39:49.733674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733653 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a394455d-d89b-4513-b82b-22db02ea07ea" containerName="payload-pre-processing" Nov 04 03:39:49.733674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733660 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a394455d-d89b-4513-b82b-22db02ea07ea" containerName="payload-pre-processing" Nov 04 03:39:49.733925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733801 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b658291f-ee09-429e-8b9f-5ddf99384ed3" containerName="payload-processing" Nov 04 03:39:49.733925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733818 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a394455d-d89b-4513-b82b-22db02ea07ea" containerName="payload-pre-processing" Nov 04 03:39:49.733925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.733833 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2950b8be-019d-4e7f-974c-ca4b9d64a404" containerName="payload-processing" Nov 04 03:39:49.737012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.736993 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.791076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.791050 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:39:49.791212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.791079 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:39:49.793444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.793414 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:39:49.841569 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.841540 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:39:49.841711 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.841589 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:39:49.842388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.842372 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77qb\" (UniqueName: \"kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.842451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.842410 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.842451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.842432 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.844528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.844509 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk"] Nov 04 03:39:49.943153 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.943127 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77qb\" (UniqueName: \"kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.943271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.943180 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.943271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.943205 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.943853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.943830 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:49.945739 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:49.945721 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:50.004848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:50.004786 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77qb\" (UniqueName: \"kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb\") pod \"maas-api-revoke-sub-d2ea8680-mlvvk\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:50.050789 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:50.050756 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:50.429534 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:50.429505 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk"] Nov 04 03:39:50.431094 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:39:50.431073 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37412413_d8ad_4154_bbb6_31c491459a05.slice/crio-a77627756d4331e0969621407ac8e643d108974044d8e3fdeb630aee077be5e3 WatchSource:0}: Error finding container a77627756d4331e0969621407ac8e643d108974044d8e3fdeb630aee077be5e3: Status 404 returned error can't find the container with id a77627756d4331e0969621407ac8e643d108974044d8e3fdeb630aee077be5e3 Nov 04 03:39:51.436854 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:51.436823 4959 generic.go:356] "Generic (PLEG): container finished" podID="37412413-d8ad-4154-bbb6-31c491459a05" containerID="55a48c670f90ec323e5ca933e057d57386eff822181d74f53bfa260da6732a9b" exitCode=0 Nov 04 03:39:51.437243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:51.436897 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" event={"ID":"37412413-d8ad-4154-bbb6-31c491459a05","Type":"ContainerDied","Data":"55a48c670f90ec323e5ca933e057d57386eff822181d74f53bfa260da6732a9b"} Nov 04 03:39:51.437243 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:51.436948 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" event={"ID":"37412413-d8ad-4154-bbb6-31c491459a05","Type":"ContainerStarted","Data":"a77627756d4331e0969621407ac8e643d108974044d8e3fdeb630aee077be5e3"} Nov 04 03:39:52.597962 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.597937 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:52.675679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.675648 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca\") pod \"37412413-d8ad-4154-bbb6-31c491459a05\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " Nov 04 03:39:52.675840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.675685 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb\" (UniqueName: \"kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb\") pod \"37412413-d8ad-4154-bbb6-31c491459a05\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " Nov 04 03:39:52.675883 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.675843 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca\") pod \"37412413-d8ad-4154-bbb6-31c491459a05\" (UID: \"37412413-d8ad-4154-bbb6-31c491459a05\") " Nov 04 03:39:52.676163 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.676140 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca" pod "37412413-d8ad-4154-bbb6-31c491459a05" (UID: "37412413-d8ad-4154-bbb6-31c491459a05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:52.676367 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.676319 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:52.677860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.677835 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca" pod "37412413-d8ad-4154-bbb6-31c491459a05" (UID: "37412413-d8ad-4154-bbb6-31c491459a05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:39:52.677978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.677959 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb" pod "37412413-d8ad-4154-bbb6-31c491459a05" (UID: "37412413-d8ad-4154-bbb6-31c491459a05"). InnerVolumeSpecName "kube-api-access-c77qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:52.777888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.777805 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37412413-d8ad-4154-bbb6-31c491459a05-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:52.777888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:52.777846 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77qb\" (UniqueName: \"kubernetes.io/projected/37412413-d8ad-4154-bbb6-31c491459a05-kube-api-access-c77qb\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:53.437314 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:53.437284 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:53.450039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:53.450005 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" Nov 04 03:39:53.450201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:53.450040 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk" event={"ID":"37412413-d8ad-4154-bbb6-31c491459a05","Type":"ContainerDied","Data":"a77627756d4331e0969621407ac8e643d108974044d8e3fdeb630aee077be5e3"} Nov 04 03:39:53.450201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:53.450088 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a77627756d4331e0969621407ac8e643d108974044d8e3fdeb630aee077be5e3" Nov 04 03:39:56.398065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.398040 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:39:56.471399 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.471369 4959 generic.go:356] "Generic (PLEG): container finished" podID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerID="5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8" exitCode=0 Nov 04 03:39:56.471600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.471429 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" Nov 04 03:39:56.471600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.471451 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" event={"ID":"3e7a5057-08aa-46d1-b41e-b9247da5524a","Type":"ContainerDied","Data":"5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8"} Nov 04 03:39:56.471600 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.471571 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q" event={"ID":"3e7a5057-08aa-46d1-b41e-b9247da5524a","Type":"ContainerDied","Data":"81ff667aa8c1009818e7772160418c1d91681eaf430c5bb093507f4d0a5b747a"} Nov 04 03:39:56.471764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.471602 4959 scope.go:122] "RemoveContainer" containerID="5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8" Nov 04 03:39:56.473246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.473227 4959 generic.go:356] "Generic (PLEG): container finished" podID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerID="5970cd707b4dc34385c075e5d60025f2c72d6e09b413833e70ea5a33c36f8b69" exitCode=0 Nov 04 03:39:56.473353 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.473255 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" event={"ID":"8aaf687b-9810-4e14-8622-57cd47096ab1","Type":"ContainerDied","Data":"5970cd707b4dc34385c075e5d60025f2c72d6e09b413833e70ea5a33c36f8b69"} Nov 04 03:39:56.481827 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.481808 4959 scope.go:122] "RemoveContainer" containerID="5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8" Nov 04 03:39:56.482128 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:39:56.482108 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8\": container with ID starting with 5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8 not found: ID does not exist" containerID="5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8" Nov 04 03:39:56.482195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.482140 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8"} err="failed to get container status \"5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8\": rpc error: code = NotFound desc = could not find container \"5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8\": container with ID starting with 5cedaa2bce60b4f49244648452b3bfd0ab51f0d886689b47fa4f1fd355b7c8a8 not found: ID does not exist" Nov 04 03:39:56.503391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.503368 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:39:56.530529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.530498 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume\") pod \"3e7a5057-08aa-46d1-b41e-b9247da5524a\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " Nov 04 03:39:56.530693 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.530674 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc\" (UniqueName: \"kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc\") pod \"3e7a5057-08aa-46d1-b41e-b9247da5524a\" (UID: \"3e7a5057-08aa-46d1-b41e-b9247da5524a\") " Nov 04 03:39:56.530831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.530809 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume" pod "3e7a5057-08aa-46d1-b41e-b9247da5524a" (UID: "3e7a5057-08aa-46d1-b41e-b9247da5524a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:56.531086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.531061 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e7a5057-08aa-46d1-b41e-b9247da5524a-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:56.532998 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.532976 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc" pod "3e7a5057-08aa-46d1-b41e-b9247da5524a" (UID: "3e7a5057-08aa-46d1-b41e-b9247da5524a"). InnerVolumeSpecName "kube-api-access-b5lkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:56.631583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.631409 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k\" (UniqueName: \"kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k\") pod \"8aaf687b-9810-4e14-8622-57cd47096ab1\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " Nov 04 03:39:56.631970 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.631947 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume\") pod \"8aaf687b-9810-4e14-8622-57cd47096ab1\" (UID: \"8aaf687b-9810-4e14-8622-57cd47096ab1\") " Nov 04 03:39:56.632717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.632294 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume" pod "8aaf687b-9810-4e14-8622-57cd47096ab1" (UID: "8aaf687b-9810-4e14-8622-57cd47096ab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:39:56.632717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.632572 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lkc\" (UniqueName: \"kubernetes.io/projected/3e7a5057-08aa-46d1-b41e-b9247da5524a-kube-api-access-b5lkc\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:56.632717 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.632595 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf687b-9810-4e14-8622-57cd47096ab1-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:56.634538 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.634509 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k" pod "8aaf687b-9810-4e14-8622-57cd47096ab1" (UID: "8aaf687b-9810-4e14-8622-57cd47096ab1"). InnerVolumeSpecName "kube-api-access-pf27k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:39:56.734128 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.734096 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf27k\" (UniqueName: \"kubernetes.io/projected/8aaf687b-9810-4e14-8622-57cd47096ab1-kube-api-access-pf27k\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:39:56.893226 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.893159 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q"] Nov 04 03:39:56.899598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:56.899569 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9dda20-7fdd45556-fb76q"] Nov 04 03:39:57.482397 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:57.482367 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" Nov 04 03:39:57.482879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:57.482536 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml" event={"ID":"8aaf687b-9810-4e14-8622-57cd47096ab1","Type":"ContainerDied","Data":"b301ab8622e5ce7226cbbfd10ce76a90fd1ecfdb3b09ce042d79cbf92587dd8b"} Nov 04 03:39:57.482879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:57.482586 4959 scope.go:122] "RemoveContainer" containerID="5970cd707b4dc34385c075e5d60025f2c72d6e09b413833e70ea5a33c36f8b69" Nov 04 03:39:57.620401 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:57.620363 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:57.665063 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:57.664994 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml"] Nov 04 03:39:57.702318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:57.702282 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9dda20-d969c96ff-2v6ml"] Nov 04 03:39:58.139081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:58.139045 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:39:58.788544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:58.788494 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:39:58.853493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:58.846156 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" path="/var/lib/kubelet/pods/3e7a5057-08aa-46d1-b41e-b9247da5524a/volumes" Nov 04 03:39:58.853493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:58.847250 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" path="/var/lib/kubelet/pods/8aaf687b-9810-4e14-8622-57cd47096ab1/volumes" Nov 04 03:39:59.792978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:39:59.792939 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:40:00.501641 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:00.501601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:00.758191 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:00.758096 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:02.205872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:02.205840 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:02.715690 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:02.715655 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:02.992546 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:02.992119 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:03.147409 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:03.147379 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5"] Nov 04 03:40:03.252408 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:03.252270 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:03.518815 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:03.518726 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:03.855650 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:03.855620 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:04.196533 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:04.196463 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:04.777018 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:04.776980 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:05.529570 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:05.529451 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:05.785955 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:05.785867 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:08.148732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.148669 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" podUID="e636a527-f551-4194-b91c-af45e9237e56" containerName="payload-processing" containerID="cri-o://1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c" gracePeriod=25 Nov 04 03:40:08.422377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.422353 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:40:08.459708 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.459680 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr\") pod \"e636a527-f551-4194-b91c-af45e9237e56\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " Nov 04 03:40:08.459892 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.459838 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume\") pod \"e636a527-f551-4194-b91c-af45e9237e56\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " Nov 04 03:40:08.459940 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.459928 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials\") pod \"e636a527-f551-4194-b91c-af45e9237e56\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " Nov 04 03:40:08.459993 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.459980 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume\") pod \"e636a527-f551-4194-b91c-af45e9237e56\" (UID: \"e636a527-f551-4194-b91c-af45e9237e56\") " Nov 04 03:40:08.460266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.460233 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume" pod "e636a527-f551-4194-b91c-af45e9237e56" (UID: "e636a527-f551-4194-b91c-af45e9237e56"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:08.460443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.460420 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume" pod "e636a527-f551-4194-b91c-af45e9237e56" (UID: "e636a527-f551-4194-b91c-af45e9237e56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:08.462245 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.462218 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr" pod "e636a527-f551-4194-b91c-af45e9237e56" (UID: "e636a527-f551-4194-b91c-af45e9237e56"). InnerVolumeSpecName "kube-api-access-jznfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:08.462357 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.462249 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials" pod "e636a527-f551-4194-b91c-af45e9237e56" (UID: "e636a527-f551-4194-b91c-af45e9237e56"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:08.561599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.561568 4959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-routing-overlay-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.561599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.561596 4959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-provider-credentials\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.561599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.561606 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e636a527-f551-4194-b91c-af45e9237e56-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.561832 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.561614 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jznfr\" (UniqueName: \"kubernetes.io/projected/e636a527-f551-4194-b91c-af45e9237e56-kube-api-access-jznfr\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:08.587566 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.587533 4959 generic.go:356] "Generic (PLEG): container finished" podID="e636a527-f551-4194-b91c-af45e9237e56" containerID="1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c" exitCode=0 Nov 04 03:40:08.587723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.587587 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" Nov 04 03:40:08.587723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.587619 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" event={"ID":"e636a527-f551-4194-b91c-af45e9237e56","Type":"ContainerDied","Data":"1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c"} Nov 04 03:40:08.587723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.587685 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5" event={"ID":"e636a527-f551-4194-b91c-af45e9237e56","Type":"ContainerDied","Data":"d262fbc83c16bf695c3106fa3a68bb25e76888de60080fd84520ac1300380ca2"} Nov 04 03:40:08.587723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.587712 4959 scope.go:122] "RemoveContainer" containerID="1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c" Nov 04 03:40:08.601446 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.601428 4959 scope.go:122] "RemoveContainer" containerID="1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c" Nov 04 03:40:08.601801 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:08.601775 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c\": container with ID starting with 1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c not found: ID does not exist" containerID="1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c" Nov 04 03:40:08.601902 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.601811 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c"} err="failed to get container status \"1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c\": rpc error: code = NotFound desc = could not find container \"1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c\": container with ID starting with 1e634ad442ebb743458c38e650496b4d6b9b8ae675f205ecd980f298bab85a2c not found: ID does not exist" Nov 04 03:40:08.754574 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.754483 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5"] Nov 04 03:40:08.758370 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.758335 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5d459a/payload-processing-external-model-e2e-extmod-b-5d459a-7755tvtn5"] Nov 04 03:40:08.799434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:08.799401 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e636a527-f551-4194-b91c-af45e9237e56" path="/var/lib/kubelet/pods/e636a527-f551-4194-b91c-af45e9237e56/volumes" Nov 04 03:40:09.792853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:09.792818 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:40:09.794384 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:09.794357 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:10.073108 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:10.073015 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:10.767847 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:10.767809 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94"] Nov 04 03:40:10.871932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:10.871898 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5"] Nov 04 03:40:10.920168 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:10.920134 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:11.264960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:11.264664 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:11.547294 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:11.547206 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:11.805861 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:11.805776 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:13.607388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:13.607356 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:14.080257 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:14.080222 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g"] Nov 04 03:40:14.080544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:14.080516 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" containerID="cri-o://ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe" gracePeriod=30 Nov 04 03:40:15.768565 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:15.768527 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" podUID="8de1183b-941b-4d68-994d-41260b0df536" containerName="payload-processing" containerID="cri-o://40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389" gracePeriod=25 Nov 04 03:40:15.872425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:15.872390 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" podUID="e820bc4a-fecd-4967-aece-47530a023c93" containerName="payload-pre-processing" containerID="cri-o://6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549" gracePeriod=25 Nov 04 03:40:15.899840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:15.899788 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:16.097255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.097229 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:40:16.158483 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.158439 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:16.166713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.166695 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:40:16.193645 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.193621 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz\" (UniqueName: \"kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz\") pod \"8de1183b-941b-4d68-994d-41260b0df536\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " Nov 04 03:40:16.193845 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.193825 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume\") pod \"8de1183b-941b-4d68-994d-41260b0df536\" (UID: \"8de1183b-941b-4d68-994d-41260b0df536\") " Nov 04 03:40:16.194158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.194137 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume" pod "8de1183b-941b-4d68-994d-41260b0df536" (UID: "8de1183b-941b-4d68-994d-41260b0df536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:16.194301 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.194264 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8de1183b-941b-4d68-994d-41260b0df536-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:16.195727 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.195707 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz" pod "8de1183b-941b-4d68-994d-41260b0df536" (UID: "8de1183b-941b-4d68-994d-41260b0df536"). InnerVolumeSpecName "kube-api-access-shkzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:16.295092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.295058 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v\" (UniqueName: \"kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v\") pod \"e820bc4a-fecd-4967-aece-47530a023c93\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " Nov 04 03:40:16.295264 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.295176 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume\") pod \"e820bc4a-fecd-4967-aece-47530a023c93\" (UID: \"e820bc4a-fecd-4967-aece-47530a023c93\") " Nov 04 03:40:16.295528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.295503 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume" pod "e820bc4a-fecd-4967-aece-47530a023c93" (UID: "e820bc4a-fecd-4967-aece-47530a023c93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:16.295607 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.295578 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shkzz\" (UniqueName: \"kubernetes.io/projected/8de1183b-941b-4d68-994d-41260b0df536-kube-api-access-shkzz\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:16.297260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.297241 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v" pod "e820bc4a-fecd-4967-aece-47530a023c93" (UID: "e820bc4a-fecd-4967-aece-47530a023c93"). InnerVolumeSpecName "kube-api-access-fzr8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:16.358914 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:16.358875 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:40:16.397166 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.397137 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e820bc4a-fecd-4967-aece-47530a023c93-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:16.397166 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.397162 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzr8v\" (UniqueName: \"kubernetes.io/projected/e820bc4a-fecd-4967-aece-47530a023c93-kube-api-access-fzr8v\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:16.636348 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.636257 4959 generic.go:356] "Generic (PLEG): container finished" podID="e820bc4a-fecd-4967-aece-47530a023c93" containerID="6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549" exitCode=0 Nov 04 03:40:16.636348 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.636303 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" Nov 04 03:40:16.636611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.636343 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" event={"ID":"e820bc4a-fecd-4967-aece-47530a023c93","Type":"ContainerDied","Data":"6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549"} Nov 04 03:40:16.636611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.636391 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5" event={"ID":"e820bc4a-fecd-4967-aece-47530a023c93","Type":"ContainerDied","Data":"e0df4e8c41c3753aa1ef49df317349105210537df87807c2dcba28165f67cef0"} Nov 04 03:40:16.636611 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.636415 4959 scope.go:122] "RemoveContainer" containerID="6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549" Nov 04 03:40:16.638012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.637987 4959 generic.go:356] "Generic (PLEG): container finished" podID="8de1183b-941b-4d68-994d-41260b0df536" containerID="40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389" exitCode=0 Nov 04 03:40:16.638128 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.638032 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" Nov 04 03:40:16.638128 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.638069 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" event={"ID":"8de1183b-941b-4d68-994d-41260b0df536","Type":"ContainerDied","Data":"40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389"} Nov 04 03:40:16.638128 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.638118 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94" event={"ID":"8de1183b-941b-4d68-994d-41260b0df536","Type":"ContainerDied","Data":"21c84ec1b43065ba62e88b89168017a487ffda15c6bfc0af84df0d3cb2ee4024"} Nov 04 03:40:16.647691 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.647660 4959 scope.go:122] "RemoveContainer" containerID="6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549" Nov 04 03:40:16.648057 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:16.648039 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549\": container with ID starting with 6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549 not found: ID does not exist" containerID="6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549" Nov 04 03:40:16.648138 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.648064 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549"} err="failed to get container status \"6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549\": rpc error: code = NotFound desc = could not find container \"6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549\": container with ID starting with 6234daa29940fcec7134db125215625b8365816e56aa4b6d2e489b1310ca1549 not found: ID does not exist" Nov 04 03:40:16.648138 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.648085 4959 scope.go:122] "RemoveContainer" containerID="40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389" Nov 04 03:40:16.656556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.656539 4959 scope.go:122] "RemoveContainer" containerID="40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389" Nov 04 03:40:16.656809 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:16.656789 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389\": container with ID starting with 40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389 not found: ID does not exist" containerID="40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389" Nov 04 03:40:16.656873 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.656816 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389"} err="failed to get container status \"40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389\": rpc error: code = NotFound desc = could not find container \"40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389\": container with ID starting with 40dc6acbbdd2a222719df5dcf047b08f40fd9ea6be9a41c976fd288df1a16389 not found: ID does not exist" Nov 04 03:40:16.834957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.834924 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94"] Nov 04 03:40:16.842616 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.842593 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d1e2ab-5bd67c8c7b-ggp94"] Nov 04 03:40:16.962785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.962710 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5"] Nov 04 03:40:16.969312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:16.969290 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d1e2ab-855dc6c94c-fdcc5"] Nov 04 03:40:17.125048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:17.125016 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:17.380481 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:17.380433 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:18.069826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:18.069776 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:18.799086 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:18.799057 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de1183b-941b-4d68-994d-41260b0df536" path="/var/lib/kubelet/pods/8de1183b-941b-4d68-994d-41260b0df536/volumes" Nov 04 03:40:18.799434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:18.799421 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e820bc4a-fecd-4967-aece-47530a023c93" path="/var/lib/kubelet/pods/e820bc4a-fecd-4967-aece-47530a023c93/volumes" Nov 04 03:40:19.333806 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.333777 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:40:19.434831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.434766 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.434831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.434800 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.434989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.434929 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.434989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.434948 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.434989 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.434964 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.435133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.434999 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.435133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435023 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.435133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435083 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.435283 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435133 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo\") pod \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\" (UID: \"3f5c4bab-9e74-4bbe-861d-edb39737dbae\") " Nov 04 03:40:19.435338 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435310 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:19.435338 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435315 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:19.435446 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435423 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:19.435544 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435519 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:19.435705 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435668 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:19.435874 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435846 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.436116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435876 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.436116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435893 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.436116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.435907 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.437254 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.437226 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:19.437351 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.437261 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:19.437410 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.437356 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "kube-api-access-tps6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:19.437821 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.437800 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo" pod "3f5c4bab-9e74-4bbe-861d-edb39737dbae" (UID: "3f5c4bab-9e74-4bbe-861d-edb39737dbae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:40:19.537460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.537419 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.537460 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.537456 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tps6n\" (UniqueName: \"kubernetes.io/projected/3f5c4bab-9e74-4bbe-861d-edb39737dbae-kube-api-access-tps6n\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.537707 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.537493 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.537707 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.537507 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.537707 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.537521 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f5c4bab-9e74-4bbe-861d-edb39737dbae-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:19.659021 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.658992 4959 generic.go:356] "Generic (PLEG): container finished" podID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerID="ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe" exitCode=0 Nov 04 03:40:19.659198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.659039 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" Nov 04 03:40:19.659198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.659079 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" event={"ID":"3f5c4bab-9e74-4bbe-861d-edb39737dbae","Type":"ContainerDied","Data":"ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe"} Nov 04 03:40:19.659198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.659128 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g" event={"ID":"3f5c4bab-9e74-4bbe-861d-edb39737dbae","Type":"ContainerDied","Data":"9a5d494ea1400ea10cbac71f6a82aad48cfdeb144ac0390151d3ccc8cffd4b0f"} Nov 04 03:40:19.659198 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.659150 4959 scope.go:122] "RemoveContainer" containerID="ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe" Nov 04 03:40:19.670209 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.670196 4959 scope.go:122] "RemoveContainer" containerID="ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe" Nov 04 03:40:19.670484 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:19.670450 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe\": container with ID starting with ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe not found: ID does not exist" containerID="ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe" Nov 04 03:40:19.670586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.670492 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe"} err="failed to get container status \"ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe\": rpc error: code = NotFound desc = could not find container \"ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe\": container with ID starting with ae2dbda0c4aa559e120edd1ae65b710668d1bd98a7fde39d3bd3be8b45bc5abe not found: ID does not exist" Nov 04 03:40:19.716538 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.716456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:19.793098 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.793067 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:40:19.826352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.826315 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g"] Nov 04 03:40:19.832525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:19.832495 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-xrc2g"] Nov 04 03:40:20.088342 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:20.088308 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd"] Nov 04 03:40:20.289643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:20.289610 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr"] Nov 04 03:40:20.603058 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:20.603021 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:20.798449 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:20.798409 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" path="/var/lib/kubelet/pods/3f5c4bab-9e74-4bbe-861d-edb39737dbae/volumes" Nov 04 03:40:21.937796 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:21.937763 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:22.220922 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:22.220830 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:23.688239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.688203 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-859b946898-skctn"] Nov 04 03:40:23.689449 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.689423 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:40:23.735506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.735459 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-cd8696ff5-4976v"] Nov 04 03:40:23.736174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736153 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" Nov 04 03:40:23.736174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736174 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736185 4959 cpu_manager.go:414] "containerMap: removing container" podUID="37412413-d8ad-4154-bbb6-31c491459a05" containerName="revoke-keys" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736190 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37412413-d8ad-4154-bbb6-31c491459a05" containerName="revoke-keys" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736200 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerName="payload-pre-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736207 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerName="payload-pre-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736241 4959 cpu_manager.go:414] "containerMap: removing container" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerName="payload-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736248 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerName="payload-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736259 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e820bc4a-fecd-4967-aece-47530a023c93" containerName="payload-pre-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736263 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e820bc4a-fecd-4967-aece-47530a023c93" containerName="payload-pre-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736270 4959 cpu_manager.go:414] "containerMap: removing container" podUID="8de1183b-941b-4d68-994d-41260b0df536" containerName="payload-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736274 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de1183b-941b-4d68-994d-41260b0df536" containerName="payload-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736281 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e636a527-f551-4194-b91c-af45e9237e56" containerName="payload-processing" Nov 04 03:40:23.736324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736285 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e636a527-f551-4194-b91c-af45e9237e56" containerName="payload-processing" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736360 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7a5057-08aa-46d1-b41e-b9247da5524a" containerName="payload-processing" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736367 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e820bc4a-fecd-4967-aece-47530a023c93" containerName="payload-pre-processing" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736372 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de1183b-941b-4d68-994d-41260b0df536" containerName="payload-processing" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736379 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e636a527-f551-4194-b91c-af45e9237e56" containerName="payload-processing" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736386 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5c4bab-9e74-4bbe-861d-edb39737dbae" containerName="istio-proxy" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736393 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="37412413-d8ad-4154-bbb6-31c491459a05" containerName="revoke-keys" Nov 04 03:40:23.736738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.736399 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf687b-9810-4e14-8622-57cd47096ab1" containerName="payload-pre-processing" Nov 04 03:40:23.741232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.741211 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.796899 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.796870 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-375c75cb8bb2\"" Nov 04 03:40:23.803303 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.803245 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd8696ff5-4976v"] Nov 04 03:40:23.811957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.811920 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428wg\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.812122 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.811964 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.812210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.812127 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.812210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.812193 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.913291 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.913255 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.913504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.913332 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.913588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.913514 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428wg\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.913588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.913544 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.913979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.913957 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.914103 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.914079 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.916110 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.916089 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:23.974958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:23.974898 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428wg\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg\") pod \"payload-processing-external-model-cd8696ff5-4976v\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:24.055526 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:24.055497 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:24.433122 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:24.433096 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd8696ff5-4976v"] Nov 04 03:40:24.434246 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:40:24.434214 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf6e2e1_0717_43a8_919a_46f87ff44be0.slice/crio-43d362eddfe7415a37c41852ef478cc8492a8379562e2828c0874a9674339225 WatchSource:0}: Error finding container 43d362eddfe7415a37c41852ef478cc8492a8379562e2828c0874a9674339225: Status 404 returned error can't find the container with id 43d362eddfe7415a37c41852ef478cc8492a8379562e2828c0874a9674339225 Nov 04 03:40:24.709393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:24.709303 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" event={"ID":"bbf6e2e1-0717-43a8-919a-46f87ff44be0","Type":"ContainerStarted","Data":"37b1895fb29845120b4b9a3184aee03e25cc67fb6da720931ea7f5a375a4b6aa"} Nov 04 03:40:24.709393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:24.709359 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" event={"ID":"bbf6e2e1-0717-43a8-919a-46f87ff44be0","Type":"ContainerStarted","Data":"43d362eddfe7415a37c41852ef478cc8492a8379562e2828c0874a9674339225"} Nov 04 03:40:24.709816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:24.709450 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:24.829278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:24.829224 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" podStartSLOduration=1.829211342 podStartE2EDuration="1.829211342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:40:24.827135379 +0000 UTC m=+1402.210563416" watchObservedRunningTime="2026-11-04 03:40:24.829211342 +0000 UTC m=+1402.212639382" Nov 04 03:40:25.088662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.088624 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" podUID="14c56253-aa80-4b21-abc7-6832d690aac8" containerName="payload-processing" containerID="cri-o://2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83" gracePeriod=25 Nov 04 03:40:25.290642 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.290596 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" podUID="568ca1fb-edf3-43af-aa30-08f3515e9bf0" containerName="payload-pre-processing" containerID="cri-o://81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95" gracePeriod=25 Nov 04 03:40:25.413956 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.413928 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:40:25.430370 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.430334 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume\") pod \"14c56253-aa80-4b21-abc7-6832d690aac8\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " Nov 04 03:40:25.430585 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.430569 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52\" (UniqueName: \"kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52\") pod \"14c56253-aa80-4b21-abc7-6832d690aac8\" (UID: \"14c56253-aa80-4b21-abc7-6832d690aac8\") " Nov 04 03:40:25.430753 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.430727 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume" pod "14c56253-aa80-4b21-abc7-6832d690aac8" (UID: "14c56253-aa80-4b21-abc7-6832d690aac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:25.430876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.430860 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c56253-aa80-4b21-abc7-6832d690aac8-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:25.432709 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.432689 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52" pod "14c56253-aa80-4b21-abc7-6832d690aac8" (UID: "14c56253-aa80-4b21-abc7-6832d690aac8"). InnerVolumeSpecName "kube-api-access-tpb52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:25.532551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.532520 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpb52\" (UniqueName: \"kubernetes.io/projected/14c56253-aa80-4b21-abc7-6832d690aac8-kube-api-access-tpb52\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:25.542527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.542460 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:40:25.593384 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.593347 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:25.633782 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.633697 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume\") pod \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " Nov 04 03:40:25.633937 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.633785 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx\" (UniqueName: \"kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx\") pod \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\" (UID: \"568ca1fb-edf3-43af-aa30-08f3515e9bf0\") " Nov 04 03:40:25.634125 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.634100 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume" pod "568ca1fb-edf3-43af-aa30-08f3515e9bf0" (UID: "568ca1fb-edf3-43af-aa30-08f3515e9bf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:25.636019 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.635993 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx" pod "568ca1fb-edf3-43af-aa30-08f3515e9bf0" (UID: "568ca1fb-edf3-43af-aa30-08f3515e9bf0"). InnerVolumeSpecName "kube-api-access-dmkzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:25.716516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.716462 4959 generic.go:356] "Generic (PLEG): container finished" podID="14c56253-aa80-4b21-abc7-6832d690aac8" containerID="2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83" exitCode=0 Nov 04 03:40:25.716961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.716541 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" Nov 04 03:40:25.716961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.716564 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" event={"ID":"14c56253-aa80-4b21-abc7-6832d690aac8","Type":"ContainerDied","Data":"2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83"} Nov 04 03:40:25.716961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.716622 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd" event={"ID":"14c56253-aa80-4b21-abc7-6832d690aac8","Type":"ContainerDied","Data":"c8160258391651004cdafb99ded42c8b84b92564955ae348a81308b79f6b284a"} Nov 04 03:40:25.716961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.716651 4959 scope.go:122] "RemoveContainer" containerID="2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83" Nov 04 03:40:25.718041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.718019 4959 generic.go:356] "Generic (PLEG): container finished" podID="568ca1fb-edf3-43af-aa30-08f3515e9bf0" containerID="81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95" exitCode=0 Nov 04 03:40:25.718135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.718063 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" Nov 04 03:40:25.718135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.718093 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" event={"ID":"568ca1fb-edf3-43af-aa30-08f3515e9bf0","Type":"ContainerDied","Data":"81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95"} Nov 04 03:40:25.718409 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.718145 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr" event={"ID":"568ca1fb-edf3-43af-aa30-08f3515e9bf0","Type":"ContainerDied","Data":"36d21be1ba428cc0258888dae3506942c3f2f4e22802f392a7e1cbd197127ac0"} Nov 04 03:40:25.727529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.727510 4959 scope.go:122] "RemoveContainer" containerID="2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83" Nov 04 03:40:25.728770 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:25.728741 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83\": container with ID starting with 2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83 not found: ID does not exist" containerID="2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83" Nov 04 03:40:25.728872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.728781 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83"} err="failed to get container status \"2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83\": rpc error: code = NotFound desc = could not find container \"2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83\": container with ID starting with 2f8b158c60f77ea2fe80bab0a7ca33b42c72f3c10adf27f351ade908510a0d83 not found: ID does not exist" Nov 04 03:40:25.728872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.728802 4959 scope.go:122] "RemoveContainer" containerID="81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95" Nov 04 03:40:25.735376 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.735353 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/568ca1fb-edf3-43af-aa30-08f3515e9bf0-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:25.735376 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.735381 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmkzx\" (UniqueName: \"kubernetes.io/projected/568ca1fb-edf3-43af-aa30-08f3515e9bf0-kube-api-access-dmkzx\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:25.737766 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.737753 4959 scope.go:122] "RemoveContainer" containerID="81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95" Nov 04 03:40:25.738045 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:25.738027 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95\": container with ID starting with 81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95 not found: ID does not exist" containerID="81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95" Nov 04 03:40:25.738094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.738053 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95"} err="failed to get container status \"81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95\": rpc error: code = NotFound desc = could not find container \"81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95\": container with ID starting with 81a24e79f0e26e185f92638e7969de608fb39daebc2886eb698fc10dec895d95 not found: ID does not exist" Nov 04 03:40:25.884167 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.884084 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd"] Nov 04 03:40:25.889694 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:25.889666 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-491fe7-f55cd5bb-n5vmd"] Nov 04 03:40:26.000804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:26.000775 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr"] Nov 04 03:40:26.008190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:26.008165 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-491fe7-85b4586559-9wjdr"] Nov 04 03:40:26.235365 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:26.235279 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:26.799790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:26.799752 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c56253-aa80-4b21-abc7-6832d690aac8" path="/var/lib/kubelet/pods/14c56253-aa80-4b21-abc7-6832d690aac8/volumes" Nov 04 03:40:26.800166 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:26.800148 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568ca1fb-edf3-43af-aa30-08f3515e9bf0" path="/var/lib/kubelet/pods/568ca1fb-edf3-43af-aa30-08f3515e9bf0/volumes" Nov 04 03:40:28.688529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:28.688456 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" containerID="cri-o://9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e" gracePeriod=25 Nov 04 03:40:28.954615 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:28.954551 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:40:29.072441 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.072409 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume\") pod \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " Nov 04 03:40:29.072647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.072516 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt\") pod \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " Nov 04 03:40:29.072647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.072546 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume\") pod \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " Nov 04 03:40:29.072647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.072583 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials\") pod \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\" (UID: \"9c5ca16b-7ce8-4c31-a15b-f3273000c867\") " Nov 04 03:40:29.072834 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.072770 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume" pod "9c5ca16b-7ce8-4c31-a15b-f3273000c867" (UID: "9c5ca16b-7ce8-4c31-a15b-f3273000c867"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:29.073070 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.073042 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume" pod "9c5ca16b-7ce8-4c31-a15b-f3273000c867" (UID: "9c5ca16b-7ce8-4c31-a15b-f3273000c867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:29.073178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.073136 4959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-routing-overlay-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:29.073178 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.073150 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c5ca16b-7ce8-4c31-a15b-f3273000c867-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:29.075132 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.075106 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt" pod "9c5ca16b-7ce8-4c31-a15b-f3273000c867" (UID: "9c5ca16b-7ce8-4c31-a15b-f3273000c867"). InnerVolumeSpecName "kube-api-access-gjpmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:29.075399 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.075375 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials" pod "9c5ca16b-7ce8-4c31-a15b-f3273000c867" (UID: "9c5ca16b-7ce8-4c31-a15b-f3273000c867"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:29.173972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.173937 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjpmt\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-kube-api-access-gjpmt\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:29.173972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.173973 4959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c5ca16b-7ce8-4c31-a15b-f3273000c867-provider-credentials\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:29.745976 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.745947 4959 generic.go:356] "Generic (PLEG): container finished" podID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerID="9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e" exitCode=0 Nov 04 03:40:29.746385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.745990 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" Nov 04 03:40:29.746385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.746031 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" event={"ID":"9c5ca16b-7ce8-4c31-a15b-f3273000c867","Type":"ContainerDied","Data":"9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e"} Nov 04 03:40:29.746385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.746085 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-859b946898-skctn" event={"ID":"9c5ca16b-7ce8-4c31-a15b-f3273000c867","Type":"ContainerDied","Data":"a53f9da3532227842cd197a0a53685805a3eee0c8b4cecae47ccc3aff54f4234"} Nov 04 03:40:29.746385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.746111 4959 scope.go:122] "RemoveContainer" containerID="9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e" Nov 04 03:40:29.756657 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.756630 4959 scope.go:122] "RemoveContainer" containerID="9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e" Nov 04 03:40:29.756959 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:29.756938 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e\": container with ID starting with 9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e not found: ID does not exist" containerID="9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e" Nov 04 03:40:29.757019 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.756969 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e"} err="failed to get container status \"9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e\": rpc error: code = NotFound desc = could not find container \"9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e\": container with ID starting with 9fc9c57c228846639967058804156f5780ec65881e7fbb62caffd211db19b95e not found: ID does not exist" Nov 04 03:40:29.914157 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.914121 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-859b946898-skctn"] Nov 04 03:40:29.964682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:29.964648 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-859b946898-skctn"] Nov 04 03:40:30.799223 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:30.799172 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" path="/var/lib/kubelet/pods/9c5ca16b-7ce8-4c31-a15b-f3273000c867/volumes" Nov 04 03:40:31.055619 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:31.055540 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:34.166644 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:34.166612 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:40:35.247646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:35.247593 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:35.721678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:35.721642 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:40:37.204734 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:37.204697 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:39.966295 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:39.966258 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:40.315133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:40.315040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:41.614244 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:41.614143 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:42.044564 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:42.044458 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l"] Nov 04 03:40:42.044724 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:42.044685 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" containerID="cri-o://873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59" gracePeriod=30 Nov 04 03:40:42.063897 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:42.063868 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm"] Nov 04 03:40:42.064076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:42.064043 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" containerID="cri-o://dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2" gracePeriod=30 Nov 04 03:40:42.151050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:42.151019 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:42.455817 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:42.455782 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:46.355411 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:46.355371 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:40:47.330794 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.330760 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:40:47.361216 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.361195 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390375 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390431 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390506 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390594 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390623 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390672 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390765 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390796 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.397518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.390823 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs\") pod \"e41fd06a-2bc4-4951-8023-a5436e88ca03\" (UID: \"e41fd06a-2bc4-4951-8023-a5436e88ca03\") " Nov 04 03:40:47.398240 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.398201 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.400659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.400619 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.402344 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.402313 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.403119 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.403086 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.403578 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.403551 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:47.403807 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.403782 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.404108 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.404086 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:47.405504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.405460 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "kube-api-access-f7ftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:47.406136 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.406115 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo" pod "e41fd06a-2bc4-4951-8023-a5436e88ca03" (UID: "e41fd06a-2bc4-4951-8023-a5436e88ca03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:40:47.436254 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.436231 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:47.491919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.491892 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.491974 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492006 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492046 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492114 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492147 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492220 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492287 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492309 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.492423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492376 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy\") pod \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\" (UID: \"a2a0c4e5-099d-42ca-ab8e-59faea597b3f\") " Nov 04 03:40:47.492423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492385 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.492608 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492433 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.492608 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492522 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:47.492824 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492801 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.492891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492867 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41fd06a-2bc4-4951-8023-a5436e88ca03-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.492891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492887 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492902 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492916 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492930 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492945 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ftj\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-kube-api-access-f7ftj\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492959 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492973 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.492987 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.493001 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.493016 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.493031 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.493271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.493046 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41fd06a-2bc4-4951-8023-a5436e88ca03-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.494190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.494166 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "kube-api-access-n9qq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:47.494392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.494374 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:40:47.494835 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.494819 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:47.494884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.494870 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy" pod "a2a0c4e5-099d-42ca-ab8e-59faea597b3f" (UID: "a2a0c4e5-099d-42ca-ab8e-59faea597b3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:40:47.594342 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.594273 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.594342 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.594302 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.594342 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.594312 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.594342 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.594325 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.594342 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.594334 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9qq6\" (UniqueName: \"kubernetes.io/projected/a2a0c4e5-099d-42ca-ab8e-59faea597b3f-kube-api-access-n9qq6\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:47.911141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.911054 4959 generic.go:356] "Generic (PLEG): container finished" podID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerID="873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59" exitCode=0 Nov 04 03:40:47.911141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.911098 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" Nov 04 03:40:47.911336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.911140 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" event={"ID":"e41fd06a-2bc4-4951-8023-a5436e88ca03","Type":"ContainerDied","Data":"873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59"} Nov 04 03:40:47.911336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.911189 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l" event={"ID":"e41fd06a-2bc4-4951-8023-a5436e88ca03","Type":"ContainerDied","Data":"64c137ab0a05cee3a54695692e27a931d447052f07b8948f41ae1f4cf45cc38f"} Nov 04 03:40:47.911336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.911217 4959 scope.go:122] "RemoveContainer" containerID="873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59" Nov 04 03:40:47.912639 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.912618 4959 generic.go:356] "Generic (PLEG): container finished" podID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerID="dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2" exitCode=0 Nov 04 03:40:47.912760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.912664 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" Nov 04 03:40:47.912760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.912698 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" event={"ID":"a2a0c4e5-099d-42ca-ab8e-59faea597b3f","Type":"ContainerDied","Data":"dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2"} Nov 04 03:40:47.912760 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.912751 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm" event={"ID":"a2a0c4e5-099d-42ca-ab8e-59faea597b3f","Type":"ContainerDied","Data":"4eaa792fd29bbc80e2857146436878d508d3a42f925472c589e8fb4d2b1c9489"} Nov 04 03:40:47.925999 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.925942 4959 scope.go:122] "RemoveContainer" containerID="873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59" Nov 04 03:40:47.926286 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:47.926266 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59\": container with ID starting with 873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59 not found: ID does not exist" containerID="873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59" Nov 04 03:40:47.926365 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.926291 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59"} err="failed to get container status \"873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59\": rpc error: code = NotFound desc = could not find container \"873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59\": container with ID starting with 873b4844f565b023f69e3f52399adf90663a33079818af9b47b86b18200e3d59 not found: ID does not exist" Nov 04 03:40:47.926365 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.926309 4959 scope.go:122] "RemoveContainer" containerID="dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2" Nov 04 03:40:47.935077 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.935059 4959 scope.go:122] "RemoveContainer" containerID="dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2" Nov 04 03:40:47.935358 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:40:47.935336 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2\": container with ID starting with dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2 not found: ID does not exist" containerID="dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2" Nov 04 03:40:47.935433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:47.935367 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2"} err="failed to get container status \"dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2\": rpc error: code = NotFound desc = could not find container \"dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2\": container with ID starting with dafed8ba97873b3363f41df7c2e516d6eff349afeed38c34e090e7621e2e81b2 not found: ID does not exist" Nov 04 03:40:48.078245 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:48.078218 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm"] Nov 04 03:40:48.084076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:48.084054 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-bd7nm"] Nov 04 03:40:48.196654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:48.196578 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l"] Nov 04 03:40:48.202450 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:48.202425 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-491fe7-openshift-default-7774566777-6957l"] Nov 04 03:40:48.798147 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:48.798113 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" path="/var/lib/kubelet/pods/a2a0c4e5-099d-42ca-ab8e-59faea597b3f/volumes" Nov 04 03:40:48.798589 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:48.798575 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" path="/var/lib/kubelet/pods/e41fd06a-2bc4-4951-8023-a5436e88ca03/volumes" Nov 04 03:40:49.110426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.110393 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk"] Nov 04 03:40:49.111070 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111053 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111072 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111085 4959 cpu_manager.go:414] "containerMap: removing container" podUID="14c56253-aa80-4b21-abc7-6832d690aac8" containerName="payload-processing" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111090 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c56253-aa80-4b21-abc7-6832d690aac8" containerName="payload-processing" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111096 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111102 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111112 4959 cpu_manager.go:414] "containerMap: removing container" podUID="568ca1fb-edf3-43af-aa30-08f3515e9bf0" containerName="payload-pre-processing" Nov 04 03:40:49.111120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111117 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568ca1fb-edf3-43af-aa30-08f3515e9bf0" containerName="payload-pre-processing" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111153 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111158 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111224 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="568ca1fb-edf3-43af-aa30-08f3515e9bf0" containerName="payload-pre-processing" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111237 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a0c4e5-099d-42ca-ab8e-59faea597b3f" containerName="istio-proxy" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111242 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41fd06a-2bc4-4951-8023-a5436e88ca03" containerName="istio-proxy" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111248 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5ca16b-7ce8-4c31-a15b-f3273000c867" containerName="payload-processing" Nov 04 03:40:49.111319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.111254 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c56253-aa80-4b21-abc7-6832d690aac8" containerName="payload-processing" Nov 04 03:40:49.114451 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.114435 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.169311 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.169285 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:40:49.211663 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.211638 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchrg\" (UniqueName: \"kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.211827 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.211674 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.211922 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.211901 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.219561 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.219539 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:40:49.219561 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.219551 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:40:49.219561 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.219540 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:40:49.223668 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.223645 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk"] Nov 04 03:40:49.313506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.313442 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.313672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.313538 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchrg\" (UniqueName: \"kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.313672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.313578 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.314185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.314154 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.316251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.316230 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.375253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.375187 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchrg\" (UniqueName: \"kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg\") pod \"maas-api-revoke-sub-de351b77-bhfvk\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.427045 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.427016 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:49.800698 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.800669 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk"] Nov 04 03:40:49.803065 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:40:49.803037 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c0c26a_10e7_45fb_b097_a0af8b83b1dd.slice/crio-564b4cea8f135e11ff629e6b30fb14e36ba89e75230e00dcc0d5532e11f28bce WatchSource:0}: Error finding container 564b4cea8f135e11ff629e6b30fb14e36ba89e75230e00dcc0d5532e11f28bce: Status 404 returned error can't find the container with id 564b4cea8f135e11ff629e6b30fb14e36ba89e75230e00dcc0d5532e11f28bce Nov 04 03:40:49.927316 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:49.927285 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" event={"ID":"26c0c26a-10e7-45fb-b097-a0af8b83b1dd","Type":"ContainerStarted","Data":"564b4cea8f135e11ff629e6b30fb14e36ba89e75230e00dcc0d5532e11f28bce"} Nov 04 03:40:50.935263 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:50.935166 4959 generic.go:356] "Generic (PLEG): container finished" podID="26c0c26a-10e7-45fb-b097-a0af8b83b1dd" containerID="9fd71ef4161949e6b0f14a13050803ea13e720b4a5b5664f01e27fd205b7cd2d" exitCode=0 Nov 04 03:40:50.935263 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:50.935217 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" event={"ID":"26c0c26a-10e7-45fb-b097-a0af8b83b1dd","Type":"ContainerDied","Data":"9fd71ef4161949e6b0f14a13050803ea13e720b4a5b5664f01e27fd205b7cd2d"} Nov 04 03:40:51.614490 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:51.613989 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:52.103105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.103078 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:52.256485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.255493 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca\") pod \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " Nov 04 03:40:52.256827 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.256801 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg\" (UniqueName: \"kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg\") pod \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " Nov 04 03:40:52.256975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.256957 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca\") pod \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\" (UID: \"26c0c26a-10e7-45fb-b097-a0af8b83b1dd\") " Nov 04 03:40:52.257686 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.257424 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca" pod "26c0c26a-10e7-45fb-b097-a0af8b83b1dd" (UID: "26c0c26a-10e7-45fb-b097-a0af8b83b1dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:52.258071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.258045 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:52.260121 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.260082 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca" pod "26c0c26a-10e7-45fb-b097-a0af8b83b1dd" (UID: "26c0c26a-10e7-45fb-b097-a0af8b83b1dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:40:52.260233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.260204 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg" pod "26c0c26a-10e7-45fb-b097-a0af8b83b1dd" (UID: "26c0c26a-10e7-45fb-b097-a0af8b83b1dd"). InnerVolumeSpecName "kube-api-access-zchrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:52.361763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.360159 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:52.361763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.360199 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zchrg\" (UniqueName: \"kubernetes.io/projected/26c0c26a-10e7-45fb-b097-a0af8b83b1dd-kube-api-access-zchrg\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:52.639627 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.639588 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:52.974547 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.974451 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" Nov 04 03:40:52.974850 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.974502 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk" event={"ID":"26c0c26a-10e7-45fb-b097-a0af8b83b1dd","Type":"ContainerDied","Data":"564b4cea8f135e11ff629e6b30fb14e36ba89e75230e00dcc0d5532e11f28bce"} Nov 04 03:40:52.974986 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:52.974920 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564b4cea8f135e11ff629e6b30fb14e36ba89e75230e00dcc0d5532e11f28bce" Nov 04 03:40:53.127831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.127795 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq"] Nov 04 03:40:53.128454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.128433 4959 cpu_manager.go:414] "containerMap: removing container" podUID="26c0c26a-10e7-45fb-b097-a0af8b83b1dd" containerName="revoke-keys" Nov 04 03:40:53.128602 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.128457 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c0c26a-10e7-45fb-b097-a0af8b83b1dd" containerName="revoke-keys" Nov 04 03:40:53.128664 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.128640 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c0c26a-10e7-45fb-b097-a0af8b83b1dd" containerName="revoke-keys" Nov 04 03:40:53.131996 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.131976 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.186738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.186712 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:40:53.186860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.186756 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:40:53.186962 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.186943 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:40:53.190899 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.190875 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq"] Nov 04 03:40:53.236754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.236678 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:40:53.274101 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.274074 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.274271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.274108 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.274271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.274255 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.407503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.404193 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.407503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.404362 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.407503 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.404403 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.413324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.408344 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.419307 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.419275 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.436321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.436281 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:53.470938 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.470908 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6\") pod \"maas-api-revoke-sub-f975cb10-8jsxq\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:53.745109 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:53.745078 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:54.119643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:54.119616 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq"] Nov 04 03:40:54.121628 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:40:54.121599 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30347b50_be86_43da_b8c2_fb6ae6c872a9.slice/crio-474db1e1b0a1082f2695fefe7e9d552a547af393fbb69b8500a52e2a1f773365 WatchSource:0}: Error finding container 474db1e1b0a1082f2695fefe7e9d552a547af393fbb69b8500a52e2a1f773365: Status 404 returned error can't find the container with id 474db1e1b0a1082f2695fefe7e9d552a547af393fbb69b8500a52e2a1f773365 Nov 04 03:40:54.987396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:54.987367 4959 generic.go:356] "Generic (PLEG): container finished" podID="30347b50-be86-43da-b8c2-fb6ae6c872a9" containerID="c3042c1c2f7aae7db1685129e432cd55e5ef00dd1598d4da4babe8a42123e112" exitCode=0 Nov 04 03:40:54.987824 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:54.987445 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" event={"ID":"30347b50-be86-43da-b8c2-fb6ae6c872a9","Type":"ContainerDied","Data":"c3042c1c2f7aae7db1685129e432cd55e5ef00dd1598d4da4babe8a42123e112"} Nov 04 03:40:54.987824 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:54.987519 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" event={"ID":"30347b50-be86-43da-b8c2-fb6ae6c872a9","Type":"ContainerStarted","Data":"474db1e1b0a1082f2695fefe7e9d552a547af393fbb69b8500a52e2a1f773365"} Nov 04 03:40:56.148020 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.147994 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:56.231513 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.231460 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca\") pod \"30347b50-be86-43da-b8c2-fb6ae6c872a9\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " Nov 04 03:40:56.231663 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.231541 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6\") pod \"30347b50-be86-43da-b8c2-fb6ae6c872a9\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " Nov 04 03:40:56.231663 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.231562 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca\") pod \"30347b50-be86-43da-b8c2-fb6ae6c872a9\" (UID: \"30347b50-be86-43da-b8c2-fb6ae6c872a9\") " Nov 04 03:40:56.231824 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.231796 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca" pod "30347b50-be86-43da-b8c2-fb6ae6c872a9" (UID: "30347b50-be86-43da-b8c2-fb6ae6c872a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:40:56.232012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.231990 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:56.233772 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.233746 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca" pod "30347b50-be86-43da-b8c2-fb6ae6c872a9" (UID: "30347b50-be86-43da-b8c2-fb6ae6c872a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:40:56.233879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.233823 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6" pod "30347b50-be86-43da-b8c2-fb6ae6c872a9" (UID: "30347b50-be86-43da-b8c2-fb6ae6c872a9"). InnerVolumeSpecName "kube-api-access-7mdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:40:56.332936 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.332913 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/30347b50-be86-43da-b8c2-fb6ae6c872a9-kube-api-access-7mdf6\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:56.333047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:56.332937 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30347b50-be86-43da-b8c2-fb6ae6c872a9-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:40:57.000971 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:57.000942 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" Nov 04 03:40:57.001161 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:57.000973 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq" event={"ID":"30347b50-be86-43da-b8c2-fb6ae6c872a9","Type":"ContainerDied","Data":"474db1e1b0a1082f2695fefe7e9d552a547af393fbb69b8500a52e2a1f773365"} Nov 04 03:40:57.001161 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:57.001018 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474db1e1b0a1082f2695fefe7e9d552a547af393fbb69b8500a52e2a1f773365" Nov 04 03:40:57.666602 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:57.666571 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:58.349782 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:58.349746 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:59.001911 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:59.001877 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:59.478158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:59.478119 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:40:59.738415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:40:59.738333 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:00.831550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:00.831515 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:01.090167 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:01.090070 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:01.438545 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:01.438454 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp"] Nov 04 03:41:01.438718 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:01.438697 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" containerID="cri-o://262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b" gracePeriod=30 Nov 04 03:41:01.544252 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:01.544202 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:02.037412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:02.037352 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:02.415457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:02.415421 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:02.673913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:02.673828 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:03.148089 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.148063 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t"] Nov 04 03:41:03.148577 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.148564 4959 cpu_manager.go:414] "containerMap: removing container" podUID="30347b50-be86-43da-b8c2-fb6ae6c872a9" containerName="revoke-keys" Nov 04 03:41:03.148646 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.148580 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30347b50-be86-43da-b8c2-fb6ae6c872a9" containerName="revoke-keys" Nov 04 03:41:03.148707 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.148683 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="30347b50-be86-43da-b8c2-fb6ae6c872a9" containerName="revoke-keys" Nov 04 03:41:03.153246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.153230 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.210051 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.210025 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t"] Nov 04 03:41:03.257201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257071 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkfdt\" (UniqueName: \"kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.257201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257140 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:41:03.257201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257156 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:41:03.257201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257155 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.257201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257173 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:41:03.257522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257214 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:41:03.257728 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.257708 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.358377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.358342 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.358586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.358523 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkfdt\" (UniqueName: \"kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.358654 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.358593 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.359187 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.359162 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.361034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.361018 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.420184 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.420116 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkfdt\" (UniqueName: \"kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt\") pod \"maas-api-revoke-sub-7e8a8d38-5db7t\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.467450 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.467422 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:03.566540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.566501 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:03.842740 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:03.842714 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t"] Nov 04 03:41:03.844162 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:03.844134 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748f68c1_3683_4285_880e_80f796f71dcf.slice/crio-fcfa7684b32df9d22e705e0760f48ee581a82f0a81dccb4ba91d8cc20aec1dab WatchSource:0}: Error finding container fcfa7684b32df9d22e705e0760f48ee581a82f0a81dccb4ba91d8cc20aec1dab: Status 404 returned error can't find the container with id fcfa7684b32df9d22e705e0760f48ee581a82f0a81dccb4ba91d8cc20aec1dab Nov 04 03:41:04.074434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.074401 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" event={"ID":"748f68c1-3683-4285-880e-80f796f71dcf","Type":"ContainerStarted","Data":"fcfa7684b32df9d22e705e0760f48ee581a82f0a81dccb4ba91d8cc20aec1dab"} Nov 04 03:41:04.210680 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.210639 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:04.740049 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.740004 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j"] Nov 04 03:41:04.743900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.743879 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.803599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.803567 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j"] Nov 04 03:41:04.874350 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.874253 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.874350 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.874319 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.874585 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.874431 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dh9\" (UniqueName: \"kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.975347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.975310 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.975552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.975366 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.975552 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.975405 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dh9\" (UniqueName: \"kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.976112 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.976070 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:04.977894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:04.977873 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:05.037244 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.037213 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dh9\" (UniqueName: \"kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9\") pod \"maas-api-revoke-sub-83cf9538-knh9j\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:05.057496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.057445 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:05.085491 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.085450 4959 generic.go:356] "Generic (PLEG): container finished" podID="748f68c1-3683-4285-880e-80f796f71dcf" containerID="9831c0faf6a5375a94beb65825d2328cabdb4ffaf4ea5be4003a14e26d881389" exitCode=0 Nov 04 03:41:05.085647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.085517 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" event={"ID":"748f68c1-3683-4285-880e-80f796f71dcf","Type":"ContainerDied","Data":"9831c0faf6a5375a94beb65825d2328cabdb4ffaf4ea5be4003a14e26d881389"} Nov 04 03:41:05.123656 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.123621 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf9557897-mngcx"] Nov 04 03:41:05.129459 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.129405 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.190233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.190204 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf9557897-mngcx"] Nov 04 03:41:05.279255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.279223 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.279640 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.279346 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.279640 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.279387 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.279640 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.279428 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtknc\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.372592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.372565 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9"] Nov 04 03:41:05.376677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.376657 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.380173 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.380108 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtknc\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.380282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.380187 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.380352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.380295 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.380352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.380345 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.380895 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.380872 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.381008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.380895 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.382859 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.382841 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.435326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.435294 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9"] Nov 04 03:41:05.441117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.441094 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j"] Nov 04 03:41:05.442176 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.442150 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:05.444994 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.444972 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtknc\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc\") pod \"payload-processing-external-model-7cf9557897-mngcx\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:05.452983 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:05.452952 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c46eb4_6a05_441e_a802_53a6b59219bd.slice/crio-2502479df917fbf764a40643b573157647fce89c4194e1fca7c6beb115055ebf WatchSource:0}: Error finding container 2502479df917fbf764a40643b573157647fce89c4194e1fca7c6beb115055ebf: Status 404 returned error can't find the container with id 2502479df917fbf764a40643b573157647fce89c4194e1fca7c6beb115055ebf Nov 04 03:41:05.481051 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.481026 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfdv\" (UniqueName: \"kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.481157 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.481069 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.481157 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.481132 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.581915 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.581882 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfdv\" (UniqueName: \"kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.582056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.581939 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.582056 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.581980 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.582638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.582615 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.584405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.584387 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.643149 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.643089 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfdv\" (UniqueName: \"kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv\") pod \"maas-api-revoke-sub-7c817be0-tqtj9\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.700389 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.700360 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:05.744668 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:05.744636 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:06.077834 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.077807 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9"] Nov 04 03:41:06.079515 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:06.079488 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5be1d9_bb1e_4688_9b5a_d9b0f8494b99.slice/crio-9643ff8ab1a989ee860b76e9d4963c62f5715d634a10adc30b1446f1b67ca703 WatchSource:0}: Error finding container 9643ff8ab1a989ee860b76e9d4963c62f5715d634a10adc30b1446f1b67ca703: Status 404 returned error can't find the container with id 9643ff8ab1a989ee860b76e9d4963c62f5715d634a10adc30b1446f1b67ca703 Nov 04 03:41:06.093223 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.093192 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" event={"ID":"01c46eb4-6a05-441e-a802-53a6b59219bd","Type":"ContainerStarted","Data":"1fe350488cc83c84801eaa1289ee543ecda8b3e397646d580fcad1306f7d5268"} Nov 04 03:41:06.093326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.093256 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" event={"ID":"01c46eb4-6a05-441e-a802-53a6b59219bd","Type":"ContainerStarted","Data":"2502479df917fbf764a40643b573157647fce89c4194e1fca7c6beb115055ebf"} Nov 04 03:41:06.094586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.094565 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" event={"ID":"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99","Type":"ContainerStarted","Data":"9643ff8ab1a989ee860b76e9d4963c62f5715d634a10adc30b1446f1b67ca703"} Nov 04 03:41:06.136980 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.136948 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf9557897-mngcx"] Nov 04 03:41:06.137627 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:06.137599 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65c2603_4b46_4dc2_8f53_02985e3a5d5c.slice/crio-c92f026119ff7b9730f196054dbf8b6580b0bdb6832f5271da5c3232b4d0a8e3 WatchSource:0}: Error finding container c92f026119ff7b9730f196054dbf8b6580b0bdb6832f5271da5c3232b4d0a8e3: Status 404 returned error can't find the container with id c92f026119ff7b9730f196054dbf8b6580b0bdb6832f5271da5c3232b4d0a8e3 Nov 04 03:41:06.213464 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.213403 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" podStartSLOduration=2.213384332 podStartE2EDuration="2.213384332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:41:06.211744131 +0000 UTC m=+1443.595172175" watchObservedRunningTime="2026-11-04 03:41:06.213384332 +0000 UTC m=+1443.596812376" Nov 04 03:41:06.224115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.224086 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:06.290066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.290041 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca\") pod \"748f68c1-3683-4285-880e-80f796f71dcf\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " Nov 04 03:41:06.290405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.290123 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt\" (UniqueName: \"kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt\") pod \"748f68c1-3683-4285-880e-80f796f71dcf\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " Nov 04 03:41:06.290405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.290216 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca\") pod \"748f68c1-3683-4285-880e-80f796f71dcf\" (UID: \"748f68c1-3683-4285-880e-80f796f71dcf\") " Nov 04 03:41:06.290628 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.290603 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca" pod "748f68c1-3683-4285-880e-80f796f71dcf" (UID: "748f68c1-3683-4285-880e-80f796f71dcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:06.292355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.292333 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt" pod "748f68c1-3683-4285-880e-80f796f71dcf" (UID: "748f68c1-3683-4285-880e-80f796f71dcf"). InnerVolumeSpecName "kube-api-access-rkfdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:06.292572 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.292551 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca" pod "748f68c1-3683-4285-880e-80f796f71dcf" (UID: "748f68c1-3683-4285-880e-80f796f71dcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:06.392169 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.392132 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.392312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.392202 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkfdt\" (UniqueName: \"kubernetes.io/projected/748f68c1-3683-4285-880e-80f796f71dcf-kube-api-access-rkfdt\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.392312 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.392221 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/748f68c1-3683-4285-880e-80f796f71dcf-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.704703 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.704673 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:41:06.763569 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.763539 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:41:06.795954 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.795925 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.795967 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796043 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796077 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796109 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796125 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796154 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796235 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796273 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo\") pod \"50b5a408-6a57-436d-a4c7-10faa9e27fda\" (UID: \"50b5a408-6a57-436d-a4c7-10faa9e27fda\") " Nov 04 03:41:06.796588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796386 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:06.796588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796457 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:06.796588 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796536 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:06.796914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796668 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.796914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796687 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.796914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796703 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.796914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796753 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:06.796914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.796876 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:06.798438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.798417 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "kube-api-access-68z47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:06.798781 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.798760 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:06.799067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.799052 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:41:06.799129 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.799069 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy" pod "50b5a408-6a57-436d-a4c7-10faa9e27fda" (UID: "50b5a408-6a57-436d-a4c7-10faa9e27fda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:06.897432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.897339 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.897432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.897370 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.897432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.897380 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.897432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.897391 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68z47\" (UniqueName: \"kubernetes.io/projected/50b5a408-6a57-436d-a4c7-10faa9e27fda-kube-api-access-68z47\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.897432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.897400 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b5a408-6a57-436d-a4c7-10faa9e27fda-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.897432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.897408 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b5a408-6a57-436d-a4c7-10faa9e27fda-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:06.916564 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.916538 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj"] Nov 04 03:41:06.922827 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:06.922800 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65"] Nov 04 03:41:07.102036 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.101995 4959 generic.go:356] "Generic (PLEG): container finished" podID="bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" containerID="2f6da64edda60036655cd804ea04e50ef87e97a477489309ee3b21441a0048b5" exitCode=0 Nov 04 03:41:07.102200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.102125 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" event={"ID":"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99","Type":"ContainerDied","Data":"2f6da64edda60036655cd804ea04e50ef87e97a477489309ee3b21441a0048b5"} Nov 04 03:41:07.103807 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.103783 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" event={"ID":"a65c2603-4b46-4dc2-8f53-02985e3a5d5c","Type":"ContainerStarted","Data":"8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b"} Nov 04 03:41:07.103953 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.103822 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" event={"ID":"a65c2603-4b46-4dc2-8f53-02985e3a5d5c","Type":"ContainerStarted","Data":"c92f026119ff7b9730f196054dbf8b6580b0bdb6832f5271da5c3232b4d0a8e3"} Nov 04 03:41:07.103953 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.103846 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:41:07.108902 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.108880 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" Nov 04 03:41:07.109025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.108908 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t" event={"ID":"748f68c1-3683-4285-880e-80f796f71dcf","Type":"ContainerDied","Data":"fcfa7684b32df9d22e705e0760f48ee581a82f0a81dccb4ba91d8cc20aec1dab"} Nov 04 03:41:07.109025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.108947 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcfa7684b32df9d22e705e0760f48ee581a82f0a81dccb4ba91d8cc20aec1dab" Nov 04 03:41:07.110640 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.110612 4959 generic.go:356] "Generic (PLEG): container finished" podID="01c46eb4-6a05-441e-a802-53a6b59219bd" containerID="1fe350488cc83c84801eaa1289ee543ecda8b3e397646d580fcad1306f7d5268" exitCode=0 Nov 04 03:41:07.110745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.110686 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" event={"ID":"01c46eb4-6a05-441e-a802-53a6b59219bd","Type":"ContainerDied","Data":"1fe350488cc83c84801eaa1289ee543ecda8b3e397646d580fcad1306f7d5268"} Nov 04 03:41:07.112398 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.112372 4959 generic.go:356] "Generic (PLEG): container finished" podID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerID="262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b" exitCode=0 Nov 04 03:41:07.112517 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.112414 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" Nov 04 03:41:07.112517 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.112438 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" event={"ID":"50b5a408-6a57-436d-a4c7-10faa9e27fda","Type":"ContainerDied","Data":"262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b"} Nov 04 03:41:07.112517 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.112500 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp" event={"ID":"50b5a408-6a57-436d-a4c7-10faa9e27fda","Type":"ContainerDied","Data":"6d5339682526da08dbe20536fdd6e604e3b07cf5567e0248399d0a3befa5b4d4"} Nov 04 03:41:07.112648 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.112524 4959 scope.go:122] "RemoveContainer" containerID="262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b" Nov 04 03:41:07.122030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.122015 4959 scope.go:122] "RemoveContainer" containerID="3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531" Nov 04 03:41:07.130608 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.130592 4959 scope.go:122] "RemoveContainer" containerID="262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b" Nov 04 03:41:07.130865 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:07.130846 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b\": container with ID starting with 262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b not found: ID does not exist" containerID="262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b" Nov 04 03:41:07.130927 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.130870 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b"} err="failed to get container status \"262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b\": rpc error: code = NotFound desc = could not find container \"262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b\": container with ID starting with 262c01660777c6d863e0448719b2f8cf7fd2f42715dbf5d648ceefa7f8a84e9b not found: ID does not exist" Nov 04 03:41:07.130927 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.130887 4959 scope.go:122] "RemoveContainer" containerID="3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531" Nov 04 03:41:07.131122 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:07.131106 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531\": container with ID starting with 3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531 not found: ID does not exist" containerID="3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531" Nov 04 03:41:07.131168 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.131126 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531"} err="failed to get container status \"3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531\": rpc error: code = NotFound desc = could not find container \"3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531\": container with ID starting with 3625af28ce3196ab05ee8b4962ff19ba0a775e4a615ece07d3aa4cb45b199531 not found: ID does not exist" Nov 04 03:41:07.189692 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.189612 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:07.341288 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.341222 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podStartSLOduration=2.3412068059999998 podStartE2EDuration="2.341206806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:41:07.339084143 +0000 UTC m=+1444.722512197" watchObservedRunningTime="2026-11-04 03:41:07.341206806 +0000 UTC m=+1444.724634845" Nov 04 03:41:07.578576 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.578540 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp"] Nov 04 03:41:07.587090 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:07.587060 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d1e2ab-openshift-default-7d44fd885-64cnp"] Nov 04 03:41:08.344042 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.344008 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:08.383436 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.383367 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:08.424773 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.424733 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9\" (UniqueName: \"kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9\") pod \"01c46eb4-6a05-441e-a802-53a6b59219bd\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " Nov 04 03:41:08.424955 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.424778 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca\") pod \"01c46eb4-6a05-441e-a802-53a6b59219bd\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " Nov 04 03:41:08.425023 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.424969 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca\") pod \"01c46eb4-6a05-441e-a802-53a6b59219bd\" (UID: \"01c46eb4-6a05-441e-a802-53a6b59219bd\") " Nov 04 03:41:08.425409 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.425363 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca" pod "01c46eb4-6a05-441e-a802-53a6b59219bd" (UID: "01c46eb4-6a05-441e-a802-53a6b59219bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:08.427347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.427316 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca" pod "01c46eb4-6a05-441e-a802-53a6b59219bd" (UID: "01c46eb4-6a05-441e-a802-53a6b59219bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:08.427650 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.427614 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9" pod "01c46eb4-6a05-441e-a802-53a6b59219bd" (UID: "01c46eb4-6a05-441e-a802-53a6b59219bd"). InnerVolumeSpecName "kube-api-access-c8dh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:08.526062 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526026 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv\" (UniqueName: \"kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv\") pod \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " Nov 04 03:41:08.526248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526145 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca\") pod \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " Nov 04 03:41:08.526318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526268 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca\") pod \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\" (UID: \"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99\") " Nov 04 03:41:08.526573 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526541 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca" pod "bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" (UID: "bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:08.526761 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526742 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:08.526831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526766 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:08.526831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526780 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8dh9\" (UniqueName: \"kubernetes.io/projected/01c46eb4-6a05-441e-a802-53a6b59219bd-kube-api-access-c8dh9\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:08.526831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.526793 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c46eb4-6a05-441e-a802-53a6b59219bd-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:08.528836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.528606 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca" pod "bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" (UID: "bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:08.528958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.528871 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv" pod "bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" (UID: "bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99"). InnerVolumeSpecName "kube-api-access-hwfdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:08.650879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.650783 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:08.650879 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.650813 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwfdv\" (UniqueName: \"kubernetes.io/projected/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99-kube-api-access-hwfdv\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:08.883496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:08.880714 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" path="/var/lib/kubelet/pods/50b5a408-6a57-436d-a4c7-10faa9e27fda/volumes" Nov 04 03:41:09.068497 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.068454 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9"] Nov 04 03:41:09.077438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.077406 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p"] Nov 04 03:41:09.166651 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.166617 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" Nov 04 03:41:09.166809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.166619 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j" event={"ID":"01c46eb4-6a05-441e-a802-53a6b59219bd","Type":"ContainerDied","Data":"2502479df917fbf764a40643b573157647fce89c4194e1fca7c6beb115055ebf"} Nov 04 03:41:09.166809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.166752 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2502479df917fbf764a40643b573157647fce89c4194e1fca7c6beb115055ebf" Nov 04 03:41:09.168206 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.168190 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" Nov 04 03:41:09.168388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.168187 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9" event={"ID":"bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99","Type":"ContainerDied","Data":"9643ff8ab1a989ee860b76e9d4963c62f5715d634a10adc30b1446f1b67ca703"} Nov 04 03:41:09.168388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:09.168368 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9643ff8ab1a989ee860b76e9d4963c62f5715d634a10adc30b1446f1b67ca703" Nov 04 03:41:10.693489 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:10.693152 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:10.947799 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:10.947699 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:11.315746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:11.315657 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:11.383115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:11.383080 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq"] Nov 04 03:41:11.861234 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:11.861176 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:12.120935 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:12.120838 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:13.905494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:13.898336 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:14.155800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:14.155690 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:15.555711 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:15.555675 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:16.354595 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:16.354557 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:41:16.384217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.384169 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" podUID="237c6545-0fa9-488c-9db2-88292db0836b" containerName="payload-processing" containerID="cri-o://9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9" gracePeriod=25 Nov 04 03:41:16.643578 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.643554 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:41:16.657771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.657743 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume\") pod \"237c6545-0fa9-488c-9db2-88292db0836b\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " Nov 04 03:41:16.657912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.657860 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd\" (UniqueName: \"kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd\") pod \"237c6545-0fa9-488c-9db2-88292db0836b\" (UID: \"237c6545-0fa9-488c-9db2-88292db0836b\") " Nov 04 03:41:16.658100 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.658075 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume" pod "237c6545-0fa9-488c-9db2-88292db0836b" (UID: "237c6545-0fa9-488c-9db2-88292db0836b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:16.658222 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.658208 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/237c6545-0fa9-488c-9db2-88292db0836b-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:16.660010 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.659988 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd" pod "237c6545-0fa9-488c-9db2-88292db0836b" (UID: "237c6545-0fa9-488c-9db2-88292db0836b"). InnerVolumeSpecName "kube-api-access-gqlxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:16.759494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:16.759433 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlxd\" (UniqueName: \"kubernetes.io/projected/237c6545-0fa9-488c-9db2-88292db0836b-kube-api-access-gqlxd\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:17.058952 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.058848 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:17.226682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.226644 4959 generic.go:356] "Generic (PLEG): container finished" podID="237c6545-0fa9-488c-9db2-88292db0836b" containerID="9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9" exitCode=0 Nov 04 03:41:17.226884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.226698 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" Nov 04 03:41:17.226884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.226710 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" event={"ID":"237c6545-0fa9-488c-9db2-88292db0836b","Type":"ContainerDied","Data":"9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9"} Nov 04 03:41:17.226884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.226765 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq" event={"ID":"237c6545-0fa9-488c-9db2-88292db0836b","Type":"ContainerDied","Data":"aeb76dbbc0aa58d4046d08789dc9f0bcea2a99f61136e983322c612d238dea8c"} Nov 04 03:41:17.226884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.226791 4959 scope.go:122] "RemoveContainer" containerID="9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9" Nov 04 03:41:17.236380 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.236361 4959 scope.go:122] "RemoveContainer" containerID="9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9" Nov 04 03:41:17.236704 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:17.236683 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9\": container with ID starting with 9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9 not found: ID does not exist" containerID="9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9" Nov 04 03:41:17.236755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.236713 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9"} err="failed to get container status \"9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9\": rpc error: code = NotFound desc = could not find container \"9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9\": container with ID starting with 9f6fe93489d2661fca5a09b9a5a76e5a603573032c4643399ca187f2858f04e9 not found: ID does not exist" Nov 04 03:41:17.393212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.393182 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq"] Nov 04 03:41:17.397900 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:17.397877 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-10f834-7bdfd44746-4h4wq"] Nov 04 03:41:18.125565 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:18.125527 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:18.813868 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:18.813825 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237c6545-0fa9-488c-9db2-88292db0836b" path="/var/lib/kubelet/pods/237c6545-0fa9-488c-9db2-88292db0836b/volumes" Nov 04 03:41:18.845345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:18.845314 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:19.260449 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260416 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm"] Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260878 4959 cpu_manager.go:414] "containerMap: removing container" podUID="748f68c1-3683-4285-880e-80f796f71dcf" containerName="revoke-keys" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260889 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748f68c1-3683-4285-880e-80f796f71dcf" containerName="revoke-keys" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260899 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" containerName="revoke-keys" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260905 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" containerName="revoke-keys" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260915 4959 cpu_manager.go:414] "containerMap: removing container" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260919 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260933 4959 cpu_manager.go:414] "containerMap: removing container" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" Nov 04 03:41:19.260947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260937 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260963 4959 cpu_manager.go:414] "containerMap: removing container" podUID="01c46eb4-6a05-441e-a802-53a6b59219bd" containerName="revoke-keys" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260969 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c46eb4-6a05-441e-a802-53a6b59219bd" containerName="revoke-keys" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260980 4959 cpu_manager.go:414] "containerMap: removing container" podUID="237c6545-0fa9-488c-9db2-88292db0836b" containerName="payload-processing" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.260986 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237c6545-0fa9-488c-9db2-88292db0836b" containerName="payload-processing" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.261049 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c46eb4-6a05-441e-a802-53a6b59219bd" containerName="revoke-keys" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.261057 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="237c6545-0fa9-488c-9db2-88292db0836b" containerName="payload-processing" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.261064 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="748f68c1-3683-4285-880e-80f796f71dcf" containerName="revoke-keys" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.261069 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" containerName="revoke-keys" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.261074 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" Nov 04 03:41:19.261321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.261081 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b5a408-6a57-436d-a4c7-10faa9e27fda" containerName="istio-proxy" Nov 04 03:41:19.264164 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.264148 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.287258 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.287219 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqv6j\" (UniqueName: \"kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.287435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.287271 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.287435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.287373 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.319866 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.319801 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:41:19.323733 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.323706 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm"] Nov 04 03:41:19.370054 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.370020 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:41:19.370054 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.370029 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:41:19.370310 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.370034 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:41:19.388190 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.388156 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.388355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.388339 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqv6j\" (UniqueName: \"kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.388411 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.388385 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.388976 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.388952 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.390905 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.390885 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.452426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.452392 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqv6j\" (UniqueName: \"kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j\") pod \"maas-api-revoke-sub-4485f952-9mknm\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.578092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.578021 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:19.952126 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:19.952094 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm"] Nov 04 03:41:19.953359 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:19.953331 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcdb0a59_bf2a_4f15_affc_d11fd286af80.slice/crio-71eb03a80f818617e2abe8d57206ff34b2ebbfd3b62ac8abf3fb19bedd7b5dfb WatchSource:0}: Error finding container 71eb03a80f818617e2abe8d57206ff34b2ebbfd3b62ac8abf3fb19bedd7b5dfb: Status 404 returned error can't find the container with id 71eb03a80f818617e2abe8d57206ff34b2ebbfd3b62ac8abf3fb19bedd7b5dfb Nov 04 03:41:20.246047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:20.245968 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" event={"ID":"bcdb0a59-bf2a-4f15-affc-d11fd286af80","Type":"ContainerStarted","Data":"71eb03a80f818617e2abe8d57206ff34b2ebbfd3b62ac8abf3fb19bedd7b5dfb"} Nov 04 03:41:20.246896 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:20.246871 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:20.662438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:20.662394 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:20.921385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:20.921296 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:21.154438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.152347 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb"] Nov 04 03:41:21.156814 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.156793 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.215174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.215099 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb"] Nov 04 03:41:21.233562 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.233524 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmws\" (UniqueName: \"kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.233732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.233710 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.233891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.233870 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.253877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.253843 4959 generic.go:356] "Generic (PLEG): container finished" podID="bcdb0a59-bf2a-4f15-affc-d11fd286af80" containerID="a4ebd2cc0f33067d071e07305677a1d6ecf7a19807ebef8456f46a5a671aa575" exitCode=0 Nov 04 03:41:21.253998 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.253893 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" event={"ID":"bcdb0a59-bf2a-4f15-affc-d11fd286af80","Type":"ContainerDied","Data":"a4ebd2cc0f33067d071e07305677a1d6ecf7a19807ebef8456f46a5a671aa575"} Nov 04 03:41:21.335034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.335003 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.335175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.335090 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.335175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.335122 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmws\" (UniqueName: \"kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.343771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.336310 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.343771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.338341 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.400643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.400620 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmws\" (UniqueName: \"kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws\") pod \"maas-api-revoke-sub-434a96c9-b8rnb\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.470622 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.470556 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:21.827620 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.827443 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:21.853067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.853040 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb"] Nov 04 03:41:21.853771 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:21.853746 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9164eb2d_bc55_4b04_92d2_5fb3b95dcfaf.slice/crio-c851122b192e58b056b6b7e026fc26914eff4605efd9a877d73350d2e7af973b WatchSource:0}: Error finding container c851122b192e58b056b6b7e026fc26914eff4605efd9a877d73350d2e7af973b: Status 404 returned error can't find the container with id c851122b192e58b056b6b7e026fc26914eff4605efd9a877d73350d2e7af973b Nov 04 03:41:21.941568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.941377 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" podUID="2aeccee1-0227-4f41-a815-0d813e660f76" containerName="main" containerID="cri-o://9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a" gracePeriod=45 Nov 04 03:41:21.946962 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.946880 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" podUID="69022d9a-0412-41c2-80f0-25cd144baedf" containerName="main" containerID="cri-o://b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0" gracePeriod=45 Nov 04 03:41:21.988229 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:21.988201 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" podUID="2aeccee1-0227-4f41-a815-0d813e660f76" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused" Nov 04 03:41:22.200355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.200331 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:41:22.235060 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.235037 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:41:22.260526 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.260493 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" event={"ID":"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf","Type":"ContainerStarted","Data":"c851122b192e58b056b6b7e026fc26914eff4605efd9a877d73350d2e7af973b"} Nov 04 03:41:22.261851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.261824 4959 generic.go:356] "Generic (PLEG): container finished" podID="2aeccee1-0227-4f41-a815-0d813e660f76" containerID="9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a" exitCode=0 Nov 04 03:41:22.261988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.261859 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" Nov 04 03:41:22.261988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.261913 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" event={"ID":"2aeccee1-0227-4f41-a815-0d813e660f76","Type":"ContainerDied","Data":"9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a"} Nov 04 03:41:22.261988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.261966 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj" event={"ID":"2aeccee1-0227-4f41-a815-0d813e660f76","Type":"ContainerDied","Data":"501ea114f47f8d366d358e694be9e29a6a184f26bc64488799ad0e23ca9ea704"} Nov 04 03:41:22.262152 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.261988 4959 scope.go:122] "RemoveContainer" containerID="9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a" Nov 04 03:41:22.263183 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.263134 4959 generic.go:356] "Generic (PLEG): container finished" podID="69022d9a-0412-41c2-80f0-25cd144baedf" containerID="b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0" exitCode=0 Nov 04 03:41:22.263183 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.263170 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" Nov 04 03:41:22.263281 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.263164 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" event={"ID":"69022d9a-0412-41c2-80f0-25cd144baedf","Type":"ContainerDied","Data":"b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0"} Nov 04 03:41:22.263323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.263287 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65" event={"ID":"69022d9a-0412-41c2-80f0-25cd144baedf","Type":"ContainerDied","Data":"c028db99b187866ea86eca675d2bd9283772756c3e747db8e37901bfcbc49279"} Nov 04 03:41:22.272738 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.272714 4959 scope.go:122] "RemoveContainer" containerID="9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a" Nov 04 03:41:22.273045 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:22.273018 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a\": container with ID starting with 9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a not found: ID does not exist" containerID="9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a" Nov 04 03:41:22.273117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.273054 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a"} err="failed to get container status \"9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a\": rpc error: code = NotFound desc = could not find container \"9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a\": container with ID starting with 9bc0c7fc1b9c9ce630d73e0a55d7f70691f521976c34d1f3b64640842930c77a not found: ID does not exist" Nov 04 03:41:22.273117 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.273070 4959 scope.go:122] "RemoveContainer" containerID="b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0" Nov 04 03:41:22.282979 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.282949 4959 scope.go:122] "RemoveContainer" containerID="b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0" Nov 04 03:41:22.283311 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:22.283269 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0\": container with ID starting with b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0 not found: ID does not exist" containerID="b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0" Nov 04 03:41:22.283369 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.283324 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0"} err="failed to get container status \"b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0\": rpc error: code = NotFound desc = could not find container \"b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0\": container with ID starting with b42a943b807ff345b61e292a62897ca165248ba63ec9cd6868950736049b03f0 not found: ID does not exist" Nov 04 03:41:22.350838 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.350809 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs\" (UniqueName: \"kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs\") pod \"69022d9a-0412-41c2-80f0-25cd144baedf\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " Nov 04 03:41:22.350985 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.350882 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home\") pod \"2aeccee1-0227-4f41-a815-0d813e660f76\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " Nov 04 03:41:22.350985 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.350978 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir\") pod \"2aeccee1-0227-4f41-a815-0d813e660f76\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " Nov 04 03:41:22.351115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351015 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq\" (UniqueName: \"kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq\") pod \"69022d9a-0412-41c2-80f0-25cd144baedf\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " Nov 04 03:41:22.351115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351047 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs\") pod \"2aeccee1-0227-4f41-a815-0d813e660f76\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " Nov 04 03:41:22.351115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351079 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache\") pod \"2aeccee1-0227-4f41-a815-0d813e660f76\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " Nov 04 03:41:22.351255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351155 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5\" (UniqueName: \"kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5\") pod \"2aeccee1-0227-4f41-a815-0d813e660f76\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " Nov 04 03:41:22.351255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351165 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home" pod "2aeccee1-0227-4f41-a815-0d813e660f76" (UID: "2aeccee1-0227-4f41-a815-0d813e660f76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.351255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351203 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir\") pod \"69022d9a-0412-41c2-80f0-25cd144baedf\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " Nov 04 03:41:22.351255 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351238 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm\") pod \"2aeccee1-0227-4f41-a815-0d813e660f76\" (UID: \"2aeccee1-0227-4f41-a815-0d813e660f76\") " Nov 04 03:41:22.351455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351279 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache\") pod \"69022d9a-0412-41c2-80f0-25cd144baedf\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " Nov 04 03:41:22.351455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351286 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir" pod "2aeccee1-0227-4f41-a815-0d813e660f76" (UID: "2aeccee1-0227-4f41-a815-0d813e660f76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.351455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351329 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm\") pod \"69022d9a-0412-41c2-80f0-25cd144baedf\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " Nov 04 03:41:22.351455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351363 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home\") pod \"69022d9a-0412-41c2-80f0-25cd144baedf\" (UID: \"69022d9a-0412-41c2-80f0-25cd144baedf\") " Nov 04 03:41:22.351455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351386 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache" pod "2aeccee1-0227-4f41-a815-0d813e660f76" (UID: "2aeccee1-0227-4f41-a815-0d813e660f76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.351723 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351699 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir" pod "69022d9a-0412-41c2-80f0-25cd144baedf" (UID: "69022d9a-0412-41c2-80f0-25cd144baedf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.351851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351832 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-tmp-dir\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.351968 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351956 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-home\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.352058 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.352048 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-tmp-dir\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.352137 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.352127 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-model-cache\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.352208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.351985 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home" pod "69022d9a-0412-41c2-80f0-25cd144baedf" (UID: "69022d9a-0412-41c2-80f0-25cd144baedf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.352278 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.352161 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache" pod "69022d9a-0412-41c2-80f0-25cd144baedf" (UID: "69022d9a-0412-41c2-80f0-25cd144baedf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.353866 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.353832 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm" pod "2aeccee1-0227-4f41-a815-0d813e660f76" (UID: "2aeccee1-0227-4f41-a815-0d813e660f76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.353991 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.353965 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5" pod "2aeccee1-0227-4f41-a815-0d813e660f76" (UID: "2aeccee1-0227-4f41-a815-0d813e660f76"). InnerVolumeSpecName "kube-api-access-xtqh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:22.354065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.353982 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq" pod "69022d9a-0412-41c2-80f0-25cd144baedf" (UID: "69022d9a-0412-41c2-80f0-25cd144baedf"). InnerVolumeSpecName "kube-api-access-hwhsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:22.354713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.354695 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm" pod "69022d9a-0412-41c2-80f0-25cd144baedf" (UID: "69022d9a-0412-41c2-80f0-25cd144baedf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:22.354919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.354878 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs" pod "69022d9a-0412-41c2-80f0-25cd144baedf" (UID: "69022d9a-0412-41c2-80f0-25cd144baedf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:22.355494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.355441 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs" pod "2aeccee1-0227-4f41-a815-0d813e660f76" (UID: "2aeccee1-0227-4f41-a815-0d813e660f76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:22.393030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.393007 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:22.452872 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.452843 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca\") pod \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " Nov 04 03:41:22.453064 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.452881 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j\" (UniqueName: \"kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j\") pod \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " Nov 04 03:41:22.453064 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.452933 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca\") pod \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\" (UID: \"bcdb0a59-bf2a-4f15-affc-d11fd286af80\") " Nov 04 03:41:22.453276 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453255 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-model-cache\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453263 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca" pod "bcdb0a59-bf2a-4f15-affc-d11fd286af80" (UID: "bcdb0a59-bf2a-4f15-affc-d11fd286af80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:22.453341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453282 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-dshm\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453297 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69022d9a-0412-41c2-80f0-25cd144baedf-home\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453310 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69022d9a-0412-41c2-80f0-25cd144baedf-tls-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453324 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwhsq\" (UniqueName: \"kubernetes.io/projected/69022d9a-0412-41c2-80f0-25cd144baedf-kube-api-access-hwhsq\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453341 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453336 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeccee1-0227-4f41-a815-0d813e660f76-tls-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453349 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtqh5\" (UniqueName: \"kubernetes.io/projected/2aeccee1-0227-4f41-a815-0d813e660f76-kube-api-access-xtqh5\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.453599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.453363 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeccee1-0227-4f41-a815-0d813e660f76-dshm\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.455871 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.455847 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j" pod "bcdb0a59-bf2a-4f15-affc-d11fd286af80" (UID: "bcdb0a59-bf2a-4f15-affc-d11fd286af80"). InnerVolumeSpecName "kube-api-access-xqv6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:22.455871 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.455848 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca" pod "bcdb0a59-bf2a-4f15-affc-d11fd286af80" (UID: "bcdb0a59-bf2a-4f15-affc-d11fd286af80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:22.554944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.554911 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.554944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.554941 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqv6j\" (UniqueName: \"kubernetes.io/projected/bcdb0a59-bf2a-4f15-affc-d11fd286af80-kube-api-access-xqv6j\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.555125 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.554954 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcdb0a59-bf2a-4f15-affc-d11fd286af80-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:22.659100 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.659022 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj"] Nov 04 03:41:22.723195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.723163 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65"] Nov 04 03:41:22.726960 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.726937 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/simulator-w1-90051a-kserve-f8566f44-bsx65"] Nov 04 03:41:22.769835 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.769802 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5"] Nov 04 03:41:22.798541 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.798512 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69022d9a-0412-41c2-80f0-25cd144baedf" path="/var/lib/kubelet/pods/69022d9a-0412-41c2-80f0-25cd144baedf/volumes" Nov 04 03:41:22.836516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.836441 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj"] Nov 04 03:41:22.840596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:22.840575 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-90051a/premium-w1-90051a-kserve-74588668f-hd8bj"] Nov 04 03:41:23.153103 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.153075 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:23.271023 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.270991 4959 generic.go:356] "Generic (PLEG): container finished" podID="9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" containerID="48e252acc98d6aa1e7a36e6dfd8c76340e64ab9f34bf0339a413cc90c70151bf" exitCode=0 Nov 04 03:41:23.271206 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.271068 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" event={"ID":"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf","Type":"ContainerDied","Data":"48e252acc98d6aa1e7a36e6dfd8c76340e64ab9f34bf0339a413cc90c70151bf"} Nov 04 03:41:23.274325 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.274295 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" event={"ID":"bcdb0a59-bf2a-4f15-affc-d11fd286af80","Type":"ContainerDied","Data":"71eb03a80f818617e2abe8d57206ff34b2ebbfd3b62ac8abf3fb19bedd7b5dfb"} Nov 04 03:41:23.274432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.274345 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71eb03a80f818617e2abe8d57206ff34b2ebbfd3b62ac8abf3fb19bedd7b5dfb" Nov 04 03:41:23.274432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.274308 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm" Nov 04 03:41:23.402351 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.402312 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:23.949463 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:23.949429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:24.166152 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.166114 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" podUID="9db5325c-c80b-4633-9d97-792b69a56bac" containerName="main" containerID="cri-o://cf535f2367a1a2894148414a13adc1fd64a798289ef15cb5d49e49f57690ea69" gracePeriod=45 Nov 04 03:41:24.170083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.170051 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" podUID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" containerName="main" containerID="cri-o://0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995" gracePeriod=45 Nov 04 03:41:24.336829 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.336792 4959 generic.go:356] "Generic (PLEG): container finished" podID="9db5325c-c80b-4633-9d97-792b69a56bac" containerID="cf535f2367a1a2894148414a13adc1fd64a798289ef15cb5d49e49f57690ea69" exitCode=0 Nov 04 03:41:24.337514 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.337487 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" event={"ID":"9db5325c-c80b-4633-9d97-792b69a56bac","Type":"ContainerDied","Data":"cf535f2367a1a2894148414a13adc1fd64a798289ef15cb5d49e49f57690ea69"} Nov 04 03:41:24.550215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.550192 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:41:24.563129 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.563101 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:24.566546 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.566529 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:41:24.707839 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.707808 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca\") pod \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " Nov 04 03:41:24.708081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.707852 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home\") pod \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " Nov 04 03:41:24.708081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708000 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca\") pod \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " Nov 04 03:41:24.708081 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708066 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws\" (UniqueName: \"kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws\") pod \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\" (UID: \"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf\") " Nov 04 03:41:24.708251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708102 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home" pod "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" (UID: "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.708251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708126 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv\" (UniqueName: \"kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv\") pod \"9db5325c-c80b-4633-9d97-792b69a56bac\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " Nov 04 03:41:24.708251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708178 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm\") pod \"9db5325c-c80b-4633-9d97-792b69a56bac\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " Nov 04 03:41:24.708251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708212 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home\") pod \"9db5325c-c80b-4633-9d97-792b69a56bac\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " Nov 04 03:41:24.708251 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708242 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache\") pod \"9db5325c-c80b-4633-9d97-792b69a56bac\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " Nov 04 03:41:24.708527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708291 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir\") pod \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " Nov 04 03:41:24.708527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708348 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs\" (UniqueName: \"kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs\") pod \"9db5325c-c80b-4633-9d97-792b69a56bac\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " Nov 04 03:41:24.708527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708373 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca" pod "9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" (UID: "9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:24.708527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708395 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache\") pod \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " Nov 04 03:41:24.708527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708438 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir\") pod \"9db5325c-c80b-4633-9d97-792b69a56bac\" (UID: \"9db5325c-c80b-4633-9d97-792b69a56bac\") " Nov 04 03:41:24.708527 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708490 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs\") pod \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " Nov 04 03:41:24.708833 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708527 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm\") pod \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " Nov 04 03:41:24.708833 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708562 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b\" (UniqueName: \"kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b\") pod \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\" (UID: \"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c\") " Nov 04 03:41:24.708833 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.708790 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home" pod "9db5325c-c80b-4633-9d97-792b69a56bac" (UID: "9db5325c-c80b-4633-9d97-792b69a56bac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.709598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.709008 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-home\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.709713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.709609 4959 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-home\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.709713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.709629 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.709713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.709270 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache" pod "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" (UID: "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.709713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.709290 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir" pod "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" (UID: "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.709713 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.709481 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache" pod "9db5325c-c80b-4633-9d97-792b69a56bac" (UID: "9db5325c-c80b-4633-9d97-792b69a56bac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.710751 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.710723 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca" pod "9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" (UID: "9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:24.711011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.710987 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir" pod "9db5325c-c80b-4633-9d97-792b69a56bac" (UID: "9db5325c-c80b-4633-9d97-792b69a56bac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.711310 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.711257 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws" pod "9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" (UID: "9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf"). InnerVolumeSpecName "kube-api-access-jvmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:24.712242 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.712218 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv" pod "9db5325c-c80b-4633-9d97-792b69a56bac" (UID: "9db5325c-c80b-4633-9d97-792b69a56bac"). InnerVolumeSpecName "kube-api-access-dhbxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:24.712337 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.712252 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs" pod "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" (UID: "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:24.712582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.712559 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b" pod "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" (UID: "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c"). InnerVolumeSpecName "kube-api-access-svp4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:24.712701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.712677 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm" pod "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" (UID: "b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.713072 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.713041 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs" pod "9db5325c-c80b-4633-9d97-792b69a56bac" (UID: "9db5325c-c80b-4633-9d97-792b69a56bac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:41:24.713709 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.713683 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm" pod "9db5325c-c80b-4633-9d97-792b69a56bac" (UID: "9db5325c-c80b-4633-9d97-792b69a56bac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:24.803361 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.803281 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aeccee1-0227-4f41-a815-0d813e660f76" path="/var/lib/kubelet/pods/2aeccee1-0227-4f41-a815-0d813e660f76/volumes" Nov 04 03:41:24.810448 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810419 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-model-cache\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810448 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810450 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-tmp-dir\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810495 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tls-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810512 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-dshm\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810527 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svp4b\" (UniqueName: \"kubernetes.io/projected/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-kube-api-access-svp4b\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810541 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810556 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmws\" (UniqueName: \"kubernetes.io/projected/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf-kube-api-access-jvmws\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810571 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbxv\" (UniqueName: \"kubernetes.io/projected/9db5325c-c80b-4633-9d97-792b69a56bac-kube-api-access-dhbxv\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810584 4959 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-dshm\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810596 4959 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9db5325c-c80b-4633-9d97-792b69a56bac-model-cache\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810619 4959 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c-tmp-dir\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:24.810685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:24.810632 4959 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9db5325c-c80b-4633-9d97-792b69a56bac-tls-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:25.343522 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.343490 4959 generic.go:356] "Generic (PLEG): container finished" podID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" containerID="0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995" exitCode=0 Nov 04 03:41:25.343977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.343531 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" Nov 04 03:41:25.343977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.343568 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" event={"ID":"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c","Type":"ContainerDied","Data":"0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995"} Nov 04 03:41:25.343977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.343605 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" event={"ID":"b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c","Type":"ContainerDied","Data":"e1ccff9619d663d8e0177211b8fcb0a5b9ac0cc9afc73d73f606e0fb919b152d"} Nov 04 03:41:25.343977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.343632 4959 scope.go:122] "RemoveContainer" containerID="0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995" Nov 04 03:41:25.345129 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.345104 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" Nov 04 03:41:25.345215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.345137 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9" event={"ID":"9db5325c-c80b-4633-9d97-792b69a56bac","Type":"ContainerDied","Data":"548209c7df6606c3f1ed768971e7742ae08420133ea11e0a24aa8af7cca88f59"} Nov 04 03:41:25.347107 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.347090 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" Nov 04 03:41:25.347197 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.347087 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb" event={"ID":"9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf","Type":"ContainerDied","Data":"c851122b192e58b056b6b7e026fc26914eff4605efd9a877d73350d2e7af973b"} Nov 04 03:41:25.347268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.347186 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c851122b192e58b056b6b7e026fc26914eff4605efd9a877d73350d2e7af973b" Nov 04 03:41:25.353877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.353862 4959 scope.go:122] "RemoveContainer" containerID="0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995" Nov 04 03:41:25.354148 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:25.354126 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995\": container with ID starting with 0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995 not found: ID does not exist" containerID="0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995" Nov 04 03:41:25.354192 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.354157 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995"} err="failed to get container status \"0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995\": rpc error: code = NotFound desc = could not find container \"0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995\": container with ID starting with 0c94b3b228121384e9f2e07b3b60d81ff269ff82c8f7695d14371f09674eb995 not found: ID does not exist" Nov 04 03:41:25.354192 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.354173 4959 scope.go:122] "RemoveContainer" containerID="cf535f2367a1a2894148414a13adc1fd64a798289ef15cb5d49e49f57690ea69" Nov 04 03:41:25.511418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.511328 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9"] Nov 04 03:41:25.513732 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.513712 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/premium-w2-af788a-kserve-7cb76777df-57gp9"] Nov 04 03:41:25.560638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.560595 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p" podUID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 04 03:41:25.625133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.625108 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p"] Nov 04 03:41:25.627371 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:25.627350 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-af788a/simulator-w2-af788a-kserve-74bf9d47bd-c5s6p"] Nov 04 03:41:26.227483 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:26.227437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:26.798613 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:26.798582 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db5325c-c80b-4633-9d97-792b69a56bac" path="/var/lib/kubelet/pods/9db5325c-c80b-4633-9d97-792b69a56bac/volumes" Nov 04 03:41:26.798984 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:26.798971 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" path="/var/lib/kubelet/pods/b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c/volumes" Nov 04 03:41:27.659509 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:27.659449 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerName="payload-processing" containerID="cri-o://d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9" gracePeriod=25 Nov 04 03:41:27.770340 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:27.770297 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerName="payload-pre-processing" containerID="cri-o://5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e" gracePeriod=25 Nov 04 03:41:28.058609 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:28.058526 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:28.124625 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:28.124591 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:29.053756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:29.053718 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:32.684434 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:32.684397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:32.960599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:32.960516 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:32.960599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:32.960578 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:34.001509 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:34.001455 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:34.442975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:34.442935 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:34.696666 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:34.696582 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:35.374199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:35.374156 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:36.060033 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:36.059998 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n"] Nov 04 03:41:36.110557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:36.110524 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f29174-l4c8n"] Nov 04 03:41:36.798670 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:36.798637 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0" path="/var/lib/kubelet/pods/c4ddcd5b-2dab-4171-9dbd-6029bb4b9ff0/volumes" Nov 04 03:41:38.123720 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:38.123676 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:38.171837 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:38.171737 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:39.313757 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.313716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:39.964446 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964416 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp"] Nov 04 03:41:39.964871 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964859 4959 cpu_manager.go:414] "containerMap: removing container" podUID="2aeccee1-0227-4f41-a815-0d813e660f76" containerName="main" Nov 04 03:41:39.964919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964873 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aeccee1-0227-4f41-a815-0d813e660f76" containerName="main" Nov 04 03:41:39.964919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964885 4959 cpu_manager.go:414] "containerMap: removing container" podUID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" containerName="main" Nov 04 03:41:39.964919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964890 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" containerName="main" Nov 04 03:41:39.964919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964896 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" containerName="revoke-keys" Nov 04 03:41:39.964919 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964901 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" containerName="revoke-keys" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964922 4959 cpu_manager.go:414] "containerMap: removing container" podUID="69022d9a-0412-41c2-80f0-25cd144baedf" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964926 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69022d9a-0412-41c2-80f0-25cd144baedf" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964935 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bcdb0a59-bf2a-4f15-affc-d11fd286af80" containerName="revoke-keys" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964939 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcdb0a59-bf2a-4f15-affc-d11fd286af80" containerName="revoke-keys" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964952 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9db5325c-c80b-4633-9d97-792b69a56bac" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.964957 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db5325c-c80b-4633-9d97-792b69a56bac" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.965024 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="69022d9a-0412-41c2-80f0-25cd144baedf" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.965030 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcdb0a59-bf2a-4f15-affc-d11fd286af80" containerName="revoke-keys" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.965038 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db5325c-c80b-4633-9d97-792b69a56bac" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.965046 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e09e3f-9b31-40cf-93ec-23ca6df4fe2c" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.965051 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aeccee1-0227-4f41-a815-0d813e660f76" containerName="main" Nov 04 03:41:39.965082 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.965056 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" containerName="revoke-keys" Nov 04 03:41:39.969961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:39.969945 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.086850 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.086818 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-10f834-openshift-default-dockercfg-4d9rk\"" Nov 04 03:41:40.133610 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.133570 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp"] Nov 04 03:41:40.163504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163455 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163673 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163569 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfr7\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163673 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163618 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163699 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163720 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163847 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163829 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163884 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163872 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.163969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163955 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.164008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.163990 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265070 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265165 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265114 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfr7\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265181 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265223 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265246 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265360 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265423 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265503 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265537 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265591 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265661 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265757 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265867 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.265969 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.265905 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.268052 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.268028 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.268159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.268144 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.330112 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.330083 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfr7\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.330641 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.330623 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token\") pod \"e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:40.582631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:40.582599 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:41.093222 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.092892 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp"] Nov 04 03:41:41.095178 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:41:41.095152 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858f8e5e_16ed_4539_8aee_a8c36d7e8a1f.slice/crio-f8a5616b1087e77c80db08947af9a5ae8f407d20465199ffee7ad78be9ffd1fc WatchSource:0}: Error finding container f8a5616b1087e77c80db08947af9a5ae8f407d20465199ffee7ad78be9ffd1fc: Status 404 returned error can't find the container with id f8a5616b1087e77c80db08947af9a5ae8f407d20465199ffee7ad78be9ffd1fc Nov 04 03:41:41.098454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.097988 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:41:41.098454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.098074 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:41:41.098454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.098108 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:41:41.137218 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.137181 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:41.446015 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.445906 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" event={"ID":"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f","Type":"ContainerStarted","Data":"483ec72b9941cb7217171f9cbeefe8e0367b1e5c19b3473125cbb5fc2de8e485"} Nov 04 03:41:41.446015 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.445958 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" event={"ID":"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f","Type":"ContainerStarted","Data":"f8a5616b1087e77c80db08947af9a5ae8f407d20465199ffee7ad78be9ffd1fc"} Nov 04 03:41:41.569553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.569500 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podStartSLOduration=2.569461549 podStartE2EDuration="2.569461549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:41:41.567345926 +0000 UTC m=+1478.950773978" watchObservedRunningTime="2026-11-04 03:41:41.569461549 +0000 UTC m=+1478.952889660" Nov 04 03:41:41.587652 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:41.587556 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:42.594504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:42.593782 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 03:41:42.594504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:42.593865 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 03:41:42.791499 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:42.786906 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:42.960269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:42.960186 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:42.960426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:42.960185 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:43.588494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:43.586869 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 03:41:43.588494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:43.586936 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 03:41:44.041326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:44.041071 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:44.603493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:44.597928 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 03:41:44.603493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:44.597989 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 03:41:44.703497 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:44.703077 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:45.017201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:45.017111 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:45.612047 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:45.612015 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 03:41:45.612631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:45.612599 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 03:41:45.794493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:45.794437 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:46.354623 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:46.354584 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:41:46.599914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:46.599875 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 03:41:46.600195 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:46.600164 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 03:41:46.679049 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:46.678959 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:47.500026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:47.499990 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:47.583836 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:47.583801 4959 patch_prober.go:29] interesting pod/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 04 03:41:47.584000 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:47.583864 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 04 03:41:47.755151 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:47.755053 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:47.982785 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:47.982760 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:41:48.021547 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.012740 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:48.075355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.075333 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:41:48.086510 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.086490 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume\") pod \"7b6c0eec-e13e-4967-aef0-78c0351f2333\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " Nov 04 03:41:48.086602 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.086544 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9\" (UniqueName: \"kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9\") pod \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " Nov 04 03:41:48.086641 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.086604 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg\" (UniqueName: \"kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg\") pod \"7b6c0eec-e13e-4967-aef0-78c0351f2333\" (UID: \"7b6c0eec-e13e-4967-aef0-78c0351f2333\") " Nov 04 03:41:48.086641 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.086630 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume\") pod \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\" (UID: \"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089\") " Nov 04 03:41:48.086907 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.086880 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume" pod "7b6c0eec-e13e-4967-aef0-78c0351f2333" (UID: "7b6c0eec-e13e-4967-aef0-78c0351f2333"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:48.087074 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.087052 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume" pod "41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" (UID: "41a0d6d2-3ac5-44c5-bb60-03d10fc0e089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:48.088865 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.088845 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg" pod "7b6c0eec-e13e-4967-aef0-78c0351f2333" (UID: "7b6c0eec-e13e-4967-aef0-78c0351f2333"). InnerVolumeSpecName "kube-api-access-v7zrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:48.088930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.088862 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9" pod "41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" (UID: "41a0d6d2-3ac5-44c5-bb60-03d10fc0e089"). InnerVolumeSpecName "kube-api-access-tt5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:48.124326 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.124291 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:48.188267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.188240 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5q9\" (UniqueName: \"kubernetes.io/projected/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-kube-api-access-tt5q9\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:48.188267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.188265 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7zrg\" (UniqueName: \"kubernetes.io/projected/7b6c0eec-e13e-4967-aef0-78c0351f2333-kube-api-access-v7zrg\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:48.188447 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.188277 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:48.188447 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.188286 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6c0eec-e13e-4967-aef0-78c0351f2333-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:48.263676 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.263639 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:48.504941 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.504911 4959 generic.go:356] "Generic (PLEG): container finished" podID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerID="d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9" exitCode=0 Nov 04 03:41:48.505135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.504965 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" Nov 04 03:41:48.505135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.504990 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" event={"ID":"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089","Type":"ContainerDied","Data":"d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9"} Nov 04 03:41:48.505135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.505061 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj" event={"ID":"41a0d6d2-3ac5-44c5-bb60-03d10fc0e089","Type":"ContainerDied","Data":"48ce2b81c7d20d922dd23db487064c3a02cd59bcc2cc315dbc49a67b2976a8e8"} Nov 04 03:41:48.505135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.505077 4959 scope.go:122] "RemoveContainer" containerID="d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9" Nov 04 03:41:48.506583 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.506555 4959 generic.go:356] "Generic (PLEG): container finished" podID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerID="5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e" exitCode=0 Nov 04 03:41:48.506706 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.506589 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" event={"ID":"7b6c0eec-e13e-4967-aef0-78c0351f2333","Type":"ContainerDied","Data":"5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e"} Nov 04 03:41:48.506706 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.506608 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" Nov 04 03:41:48.506706 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.506631 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5" event={"ID":"7b6c0eec-e13e-4967-aef0-78c0351f2333","Type":"ContainerDied","Data":"647e98f84d49b9f5cc9403e4f11808fa5570e7cb7235ee3c24eb9f3c79fbbce9"} Nov 04 03:41:48.517553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.517531 4959 scope.go:122] "RemoveContainer" containerID="d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9" Nov 04 03:41:48.517857 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:48.517838 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9\": container with ID starting with d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9 not found: ID does not exist" containerID="d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9" Nov 04 03:41:48.517931 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.517863 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9"} err="failed to get container status \"d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9\": rpc error: code = NotFound desc = could not find container \"d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9\": container with ID starting with d9b8eed29ac9a377b3fa2ad022e39971e4a5836f17304ce8ee74de9ebed758a9 not found: ID does not exist" Nov 04 03:41:48.517931 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.517882 4959 scope.go:122] "RemoveContainer" containerID="5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e" Nov 04 03:41:48.529156 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.529133 4959 scope.go:122] "RemoveContainer" containerID="5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e" Nov 04 03:41:48.529602 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:48.529580 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e\": container with ID starting with 5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e not found: ID does not exist" containerID="5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e" Nov 04 03:41:48.529716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.529607 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e"} err="failed to get container status \"5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e\": rpc error: code = NotFound desc = could not find container \"5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e\": container with ID starting with 5c6fb43fb4c8b36997e80b4a4f98437bd2b04a376ea37bf2e7211ef77e361c1e not found: ID does not exist" Nov 04 03:41:48.588232 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.588205 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:48.588530 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.588507 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:48.589199 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.589182 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:48.603705 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.603675 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:48.678426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.678398 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj"] Nov 04 03:41:48.684135 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.684112 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-af788a-89cdc6c6d-7wjwj"] Nov 04 03:41:48.798956 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.798882 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" path="/var/lib/kubelet/pods/41a0d6d2-3ac5-44c5-bb60-03d10fc0e089/volumes" Nov 04 03:41:48.799313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.799161 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5"] Nov 04 03:41:48.799313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.799185 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-af788a-5fd4f9b6c-rx9n5"] Nov 04 03:41:48.859373 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:48.859345 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:50.129083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:50.128988 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:50.217784 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:50.217751 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:41:50.676506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:50.674099 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:50.799374 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:50.799338 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" path="/var/lib/kubelet/pods/7b6c0eec-e13e-4967-aef0-78c0351f2333/volumes" Nov 04 03:41:51.870185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:51.870136 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:52.118977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:52.118946 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:53.264679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:53.264647 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:53.512409 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:53.512373 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:53.847752 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:53.847716 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29"] Nov 04 03:41:53.848266 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:53.848232 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" containerID="cri-o://a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2" gracePeriod=30 Nov 04 03:41:53.990495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:53.984932 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:54.239443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:54.239355 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:54.348282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:54.348248 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp"] Nov 04 03:41:54.348737 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:54.348609 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" containerID="cri-o://483ec72b9941cb7217171f9cbeefe8e0367b1e5c19b3473125cbb5fc2de8e485" gracePeriod=30 Nov 04 03:41:54.543066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:54.542983 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:54.962443 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:54.962406 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:55.477201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:55.477165 4959 patch_prober.go:29] interesting pod/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:41:55.477601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:55.477232 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:41:58.124076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:58.124034 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:41:58.443035 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:58.442924 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:41:59.105655 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.105628 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:41:59.239973 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.239884 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.239973 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.239944 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.239989 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240028 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240096 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240170 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240273 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240284 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240351 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240357 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240438 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy\") pod \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\" (UID: \"4cd864b5-ecec-4876-9d89-42083cdfd4cc\") " Nov 04 03:41:59.240568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240197 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240600 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240642 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240890 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240908 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240921 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240934 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.241591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.240948 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.242559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.242523 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "kube-api-access-278px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:59.242679 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.242642 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:59.243167 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.243143 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.243252 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.243150 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo" pod "4cd864b5-ecec-4876-9d89-42083cdfd4cc" (UID: "4cd864b5-ecec-4876-9d89-42083cdfd4cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:41:59.341756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.341724 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.341756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.341752 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278px\" (UniqueName: \"kubernetes.io/projected/4cd864b5-ecec-4876-9d89-42083cdfd4cc-kube-api-access-278px\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.341756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.341763 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.341990 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.341771 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd864b5-ecec-4876-9d89-42083cdfd4cc-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.585378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.585350 4959 generic.go:356] "Generic (PLEG): container finished" podID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerID="483ec72b9941cb7217171f9cbeefe8e0367b1e5c19b3473125cbb5fc2de8e485" exitCode=0 Nov 04 03:41:59.585529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.585417 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" event={"ID":"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f","Type":"ContainerDied","Data":"483ec72b9941cb7217171f9cbeefe8e0367b1e5c19b3473125cbb5fc2de8e485"} Nov 04 03:41:59.587272 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.587251 4959 generic.go:356] "Generic (PLEG): container finished" podID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerID="a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2" exitCode=0 Nov 04 03:41:59.587386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.587284 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" event={"ID":"4cd864b5-ecec-4876-9d89-42083cdfd4cc","Type":"ContainerDied","Data":"a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2"} Nov 04 03:41:59.587386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.587314 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" event={"ID":"4cd864b5-ecec-4876-9d89-42083cdfd4cc","Type":"ContainerDied","Data":"9b47ee19a279696d90012304544ae365e344eb73e192badcc8bbc007657143dd"} Nov 04 03:41:59.587386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.587338 4959 scope.go:122] "RemoveContainer" containerID="a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2" Nov 04 03:41:59.587386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.587348 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29" Nov 04 03:41:59.589568 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.589549 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:41:59.596691 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.596676 4959 scope.go:122] "RemoveContainer" containerID="0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397" Nov 04 03:41:59.604983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.604967 4959 scope.go:122] "RemoveContainer" containerID="a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2" Nov 04 03:41:59.605223 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:59.605207 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2\": container with ID starting with a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2 not found: ID does not exist" containerID="a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2" Nov 04 03:41:59.605270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.605231 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2"} err="failed to get container status \"a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2\": rpc error: code = NotFound desc = could not find container \"a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2\": container with ID starting with a115a69fbfa8e05883bb3d1e1f7bbfb9863b37485337ea0f2d925fba6884f7e2 not found: ID does not exist" Nov 04 03:41:59.605270 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.605247 4959 scope.go:122] "RemoveContainer" containerID="0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397" Nov 04 03:41:59.606140 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:41:59.605463 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397\": container with ID starting with 0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397 not found: ID does not exist" containerID="0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397" Nov 04 03:41:59.606221 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.606151 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397"} err="failed to get container status \"0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397\": rpc error: code = NotFound desc = could not find container \"0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397\": container with ID starting with 0efa565489b103bec334e6e305c60e7560807610c99aec40bd5c994157aa8397 not found: ID does not exist" Nov 04 03:41:59.644066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644044 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644151 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644077 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644151 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644098 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644262 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644152 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644262 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644224 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644262 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644254 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644316 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.644405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644345 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644405 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644396 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644451 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7\") pod \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\" (UID: \"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f\") " Nov 04 03:41:59.644598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644535 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.644598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644565 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.644750 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644595 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:41:59.644797 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644753 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.644988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.644960 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.645196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.645064 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.645196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.645158 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.645196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.645173 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.645196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.645187 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.646696 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.646673 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "kube-api-access-nmfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:59.646786 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.646704 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:41:59.647002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.646981 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:41:59.647002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.646997 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token" pod "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" (UID: "858f8e5e-16ed-4539-8aee-a8c36d7e8a1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:41:59.745765 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.745740 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.745904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.745770 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.745904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.745785 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfr7\" (UniqueName: \"kubernetes.io/projected/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-kube-api-access-nmfr7\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.745904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.745799 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:41:59.777392 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.777321 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29"] Nov 04 03:41:59.803911 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:41:59.803881 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-af788a-openshift-default-69fcfbcd8c-qdb29"] Nov 04 03:42:00.595390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:00.595352 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" event={"ID":"858f8e5e-16ed-4539-8aee-a8c36d7e8a1f","Type":"ContainerDied","Data":"f8a5616b1087e77c80db08947af9a5ae8f407d20465199ffee7ad78be9ffd1fc"} Nov 04 03:42:00.595816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:00.595439 4959 scope.go:122] "RemoveContainer" containerID="483ec72b9941cb7217171f9cbeefe8e0367b1e5c19b3473125cbb5fc2de8e485" Nov 04 03:42:00.595816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:00.595374 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp" Nov 04 03:42:00.764913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:00.764879 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp"] Nov 04 03:42:00.798644 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:00.798613 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" path="/var/lib/kubelet/pods/4cd864b5-ecec-4876-9d89-42083cdfd4cc/volumes" Nov 04 03:42:00.815382 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:00.815352 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-10f834-openshift-default-65d66744fc-b2vpp"] Nov 04 03:42:01.047348 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:01.047236 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn"] Nov 04 03:42:01.051763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:01.051739 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-653cbc1a-8fvtn"] Nov 04 03:42:01.183394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:01.183362 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:02.798229 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:02.798194 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05af6cd7-a401-4ed8-a220-3d4e45480d1c" path="/var/lib/kubelet/pods/05af6cd7-a401-4ed8-a220-3d4e45480d1c/volumes" Nov 04 03:42:02.798647 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:02.798633 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" path="/var/lib/kubelet/pods/858f8e5e-16ed-4539-8aee-a8c36d7e8a1f/volumes" Nov 04 03:42:02.938094 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:02.938062 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:42:02.941215 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:02.941195 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:42:02.944675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:02.944656 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:42:02.947958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:02.947934 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:42:03.360881 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:03.360845 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:03.890426 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:03.890390 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:03.890426 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:03.890420 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:03.891101 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:03.890526 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:42:04.390505537 +0000 UTC m=+1501.773933575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:04.394934 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:04.394902 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:04.394934 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:04.394930 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:04.395143 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:04.394982 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:42:05.394968332 +0000 UTC m=+1502.778396351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:05.404829 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:05.404798 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:05.404829 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:05.404822 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:05.405267 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:05.404871 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:42:07.404857787 +0000 UTC m=+1504.788285805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:07.425516 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:07.425462 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:07.425516 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:07.425511 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:07.425978 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:07.425563 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:42:11.425549245 +0000 UTC m=+1508.808977263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:08.066002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:08.065969 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:08.123918 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:08.123878 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:42:08.371576 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:08.371533 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:09.821157 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:09.821101 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:10.083146 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:10.083112 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:11.468829 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:11.468798 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:11.468829 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:11.468823 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:11.469253 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:11.468879 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:42:19.468865018 +0000 UTC m=+1516.852293037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:13.029599 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:13.029563 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:13.048174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:13.048147 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn"] Nov 04 03:42:13.097658 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:13.097633 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eea5b7af-9lxnn"] Nov 04 03:42:14.798299 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:14.798266 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd04779-e52d-4bf6-ad68-dfac94bdde16" path="/var/lib/kubelet/pods/7bd04779-e52d-4bf6-ad68-dfac94bdde16/volumes" Nov 04 03:42:16.353936 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:16.353901 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:42:17.995973 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:17.995940 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:18.123957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:18.123918 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:42:19.551145 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:19.551114 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:19.551145 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:19.551139 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:19.551588 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:19.551194 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:42:35.551177143 +0000 UTC m=+1532.934605175 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:22.747883 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:22.747846 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:24.793067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:24.793030 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:42:27.524509 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:27.524456 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:32.441944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:32.441908 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:34.793708 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:34.793662 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:42:35.626581 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:35.626552 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:42:35.626581 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:35.626576 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:42:35.626865 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:35.626633 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:43:07.626619235 +0000 UTC m=+1565.010047254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:42:37.406027 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:37.405997 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:41.563757 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:41.563716 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:42.137484 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:42.137434 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:44.793033 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:44.793001 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:42:46.354197 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:42:46.354160 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:42:46.936945 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:46.936883 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:51.809529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:51.809497 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:42:54.793439 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:54.793400 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:42:56.724115 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:42:56.724077 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:01.521002 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:01.520961 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:01.822210 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:01.822125 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:03.218956 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:03.218926 4959 scope.go:122] "RemoveContainer" containerID="8e92c40f3f117b784404dbc0dcbca443736c5a7f6061368a19d3a6c8d339a687" Nov 04 03:43:03.227991 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:03.227963 4959 scope.go:122] "RemoveContainer" containerID="4e051b9e75187d8ed1193aec4d7afd87a04ce7752068089e58601ea40fe31fe8" Nov 04 03:43:03.355627 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:03.355596 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:04.793868 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:04.793826 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:43:06.420675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:06.420640 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:07.384677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:07.384646 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:07.627441 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:07.627397 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:07.664000 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:43:07.663934 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:43:07.664000 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:43:07.663957 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:43:07.664168 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:43:07.664015 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:44:11.664000346 +0000 UTC m=+1629.047428365 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:43:11.218062 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:11.218014 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:14.796864 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:14.796764 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:43:16.162846 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:16.162808 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:16.353720 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:43:16.353686 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:43:21.126882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:21.126843 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:24.793360 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:24.793317 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:43:26.053204 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:26.053175 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:30.960271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:30.960233 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:34.793265 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:34.793230 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:43:35.844841 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:35.844803 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:40.729074 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:40.729026 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:43.048621 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:43.048586 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q"] Nov 04 03:43:43.099200 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:43.099168 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3884c08-dbv9q"] Nov 04 03:43:43.471001 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:43.470957 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:44.793265 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:44.793231 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:43:44.798181 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:44.798156 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83efd1f1-28e0-4b79-862e-8ec83bfe3e8b" path="/var/lib/kubelet/pods/83efd1f1-28e0-4b79-862e-8ec83bfe3e8b/volumes" Nov 04 03:43:45.629866 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:45.629832 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:46.354160 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:43:46.354127 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:43:48.793840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:48.793806 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:43:50.532030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:50.531995 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:53.058244 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:53.058208 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl"] Nov 04 03:43:53.108551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:53.108522 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3529b6f-jplfl"] Nov 04 03:43:54.797848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:54.797814 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b5cc2e-25df-4e48-a334-987d9aa3cd03" path="/var/lib/kubelet/pods/07b5cc2e-25df-4e48-a334-987d9aa3cd03/volumes" Nov 04 03:43:55.444464 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:55.444429 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:43:58.793993 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:43:58.793955 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:44:00.381400 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:00.381359 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:01.905604 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:01.905566 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:03.252894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:03.252855 4959 scope.go:122] "RemoveContainer" containerID="472c72ed8e02b74ca63806acb1ee395557a223a596757dc6b6f7285d3b68cdba" Nov 04 03:44:05.255039 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:05.255001 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:08.793880 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:08.793836 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:44:10.143551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:10.143506 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:11.300787 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:11.300756 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:11.715337 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:44:11.715301 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:44:11.715337 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:44:11.715330 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:44:11.715601 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:44:11.715387 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:46:13.715373961 +0000 UTC m=+1751.098801993 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:44:12.491891 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:12.491851 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:15.166670 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:15.166632 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:16.354564 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:44:16.354520 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:44:18.794455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:18.794422 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:44:20.068322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:20.068287 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:24.834977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:24.834948 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:28.793934 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:28.793898 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:44:29.623122 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:29.623084 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:34.526291 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:34.526244 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:38.803004 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:38.802961 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:44:39.317781 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:39.317740 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:44.141214 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:44.141126 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:46.354025 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:44:46.353986 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:44:48.794262 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:48.794222 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:44:48.944378 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:48.944335 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:53.051418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:53.051380 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk"] Nov 04 03:44:53.101559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:53.101524 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea8680-mlvvk"] Nov 04 03:44:53.728171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:53.728127 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:54.797916 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:54.797880 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37412413-d8ad-4154-bbb6-31c491459a05" path="/var/lib/kubelet/pods/37412413-d8ad-4154-bbb6-31c491459a05/volumes" Nov 04 03:44:58.535007 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:58.534973 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:44:58.793916 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:44:58.793832 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:00.183043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183007 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4"] Nov 04 03:45:00.183693 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183676 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerName="payload-pre-processing" Nov 04 03:45:00.183747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183696 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerName="payload-pre-processing" Nov 04 03:45:00.183747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183714 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" Nov 04 03:45:00.183747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183721 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" Nov 04 03:45:00.183860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183770 4959 cpu_manager.go:414] "containerMap: removing container" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" Nov 04 03:45:00.183860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183778 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" Nov 04 03:45:00.183860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183812 4959 cpu_manager.go:414] "containerMap: removing container" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerName="payload-processing" Nov 04 03:45:00.183860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183819 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerName="payload-processing" Nov 04 03:45:00.183983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183920 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a0d6d2-3ac5-44c5-bb60-03d10fc0e089" containerName="payload-processing" Nov 04 03:45:00.183983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183933 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6c0eec-e13e-4967-aef0-78c0351f2333" containerName="payload-pre-processing" Nov 04 03:45:00.183983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183942 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" Nov 04 03:45:00.183983 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.183955 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="858f8e5e-16ed-4539-8aee-a8c36d7e8a1f" containerName="istio-proxy" Nov 04 03:45:00.187389 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.187370 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf"] Nov 04 03:45:00.187514 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.187495 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.187790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.187769 4959 cpu_manager.go:414] "containerMap: removing container" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" Nov 04 03:45:00.187790 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.187787 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" Nov 04 03:45:00.187984 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.187960 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd864b5-ecec-4876-9d89-42083cdfd4cc" containerName="istio-proxy" Nov 04 03:45:00.190827 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.190807 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2"] Nov 04 03:45:00.190951 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.190911 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.194260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.194226 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.204898 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.204872 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.205032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.204909 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpk8f\" (UniqueName: \"kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.205032 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.204937 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.205159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.205093 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdbsq\" (UniqueName: \"kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.205213 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.205195 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.205275 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.205264 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp\") pod \"maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.240748 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.240706 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:45:00.240951 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.240837 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:45:00.240951 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.240862 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:45:00.293933 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.293886 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4"] Nov 04 03:45:00.297084 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.297046 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf"] Nov 04 03:45:00.299953 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.299921 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2"] Nov 04 03:45:00.305965 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.305931 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.306141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.305976 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpk8f\" (UniqueName: \"kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.306141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.306002 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.306141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.306057 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdbsq\" (UniqueName: \"kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.306141 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.306121 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.306350 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.306173 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp\") pod \"maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.307068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.307043 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.307944 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.307916 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.308116 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.308095 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.368703 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.368674 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp\") pod \"maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.369036 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.369008 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpk8f\" (UniqueName: \"kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f\") pod \"maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.369394 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.369374 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdbsq\" (UniqueName: \"kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.502432 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.502341 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:00.509374 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.509349 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:00.516429 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.516399 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:00.887889 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.887860 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4"] Nov 04 03:45:00.889724 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:45:00.889699 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5c9294_9c7d_42c8_9934_77a8dd8636d0.slice/crio-620bbec0a7a5f9e1c4fdb798e69e63570eec52a34839c214fd70ac82f9aee8d5 WatchSource:0}: Error finding container 620bbec0a7a5f9e1c4fdb798e69e63570eec52a34839c214fd70ac82f9aee8d5: Status 404 returned error can't find the container with id 620bbec0a7a5f9e1c4fdb798e69e63570eec52a34839c214fd70ac82f9aee8d5 Nov 04 03:45:00.938038 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.938007 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf"] Nov 04 03:45:00.939793 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:45:00.939748 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62c0ba8_77aa_4948_a872_29baa270c56d.slice/crio-3f8b81b76d22bc94e554c378ced4dcdbd5e20c9a9bec3d5729a749221f6039fd WatchSource:0}: Error finding container 3f8b81b76d22bc94e554c378ced4dcdbd5e20c9a9bec3d5729a749221f6039fd: Status 404 returned error can't find the container with id 3f8b81b76d22bc94e554c378ced4dcdbd5e20c9a9bec3d5729a749221f6039fd Nov 04 03:45:00.940438 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:00.940417 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2"] Nov 04 03:45:00.941643 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:45:00.941618 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bdfe44_4638_4d2c_87f4_a69f1eb22fa7.slice/crio-e20550ac5629d08301d351ac5ca420523fbe148af622bbce31a2be716e676083 WatchSource:0}: Error finding container e20550ac5629d08301d351ac5ca420523fbe148af622bbce31a2be716e676083: Status 404 returned error can't find the container with id e20550ac5629d08301d351ac5ca420523fbe148af622bbce31a2be716e676083 Nov 04 03:45:01.577067 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.577029 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerStarted","Data":"c125dd2422fb1931385a9820fe53f3bab9c8dfd42809b084cbca61c752a48cc9"} Nov 04 03:45:01.577542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.577101 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerStarted","Data":"3f8b81b76d22bc94e554c378ced4dcdbd5e20c9a9bec3d5729a749221f6039fd"} Nov 04 03:45:01.578748 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.578712 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" event={"ID":"ab5c9294-9c7d-42c8-9934-77a8dd8636d0","Type":"ContainerStarted","Data":"3bb190ac884a8ed0851f0182c31788e8a8fac91c5ae15bab5536035015b85a7b"} Nov 04 03:45:01.578877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.578772 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" event={"ID":"ab5c9294-9c7d-42c8-9934-77a8dd8636d0","Type":"ContainerStarted","Data":"620bbec0a7a5f9e1c4fdb798e69e63570eec52a34839c214fd70ac82f9aee8d5"} Nov 04 03:45:01.580491 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.580448 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerStarted","Data":"0360b51df75733978d874834b116358fb91d3c4002ba559282199248d85adf4a"} Nov 04 03:45:01.580612 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.580514 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerStarted","Data":"e20550ac5629d08301d351ac5ca420523fbe148af622bbce31a2be716e676083"} Nov 04 03:45:01.696515 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.696451 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" podStartSLOduration=1.696438091 podStartE2EDuration="1.696438091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:45:01.692559096 +0000 UTC m=+1679.075987136" watchObservedRunningTime="2026-11-04 03:45:01.696438091 +0000 UTC m=+1679.079866125" Nov 04 03:45:01.812354 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.812305 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" podStartSLOduration=1.8122903670000001 podStartE2EDuration="1.812290367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:45:01.809002212 +0000 UTC m=+1679.192430252" watchObservedRunningTime="2026-11-04 03:45:01.812290367 +0000 UTC m=+1679.195718418" Nov 04 03:45:01.926173 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:01.926109 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" podStartSLOduration=1.926096357 podStartE2EDuration="1.926096357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:45:01.925619329 +0000 UTC m=+1679.309047370" watchObservedRunningTime="2026-11-04 03:45:01.926096357 +0000 UTC m=+1679.309524396" Nov 04 03:45:02.222523 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.222400 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:02.588789 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.588754 4959 generic.go:356] "Generic (PLEG): container finished" podID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerID="c125dd2422fb1931385a9820fe53f3bab9c8dfd42809b084cbca61c752a48cc9" exitCode=60 Nov 04 03:45:02.589225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.588836 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerDied","Data":"c125dd2422fb1931385a9820fe53f3bab9c8dfd42809b084cbca61c752a48cc9"} Nov 04 03:45:02.589225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.589074 4959 scope.go:122] "RemoveContainer" containerID="c125dd2422fb1931385a9820fe53f3bab9c8dfd42809b084cbca61c752a48cc9" Nov 04 03:45:02.590555 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.590528 4959 generic.go:356] "Generic (PLEG): container finished" podID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerID="3bb190ac884a8ed0851f0182c31788e8a8fac91c5ae15bab5536035015b85a7b" exitCode=60 Nov 04 03:45:02.590664 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.590610 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" event={"ID":"ab5c9294-9c7d-42c8-9934-77a8dd8636d0","Type":"ContainerDied","Data":"3bb190ac884a8ed0851f0182c31788e8a8fac91c5ae15bab5536035015b85a7b"} Nov 04 03:45:02.590838 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.590821 4959 scope.go:122] "RemoveContainer" containerID="3bb190ac884a8ed0851f0182c31788e8a8fac91c5ae15bab5536035015b85a7b" Nov 04 03:45:02.592217 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.592187 4959 generic.go:356] "Generic (PLEG): container finished" podID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerID="0360b51df75733978d874834b116358fb91d3c4002ba559282199248d85adf4a" exitCode=60 Nov 04 03:45:02.592305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.592248 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerDied","Data":"0360b51df75733978d874834b116358fb91d3c4002ba559282199248d85adf4a"} Nov 04 03:45:02.592532 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:02.592516 4959 scope.go:122] "RemoveContainer" containerID="0360b51df75733978d874834b116358fb91d3c4002ba559282199248d85adf4a" Nov 04 03:45:03.278071 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.277994 4959 scope.go:122] "RemoveContainer" containerID="f1cc4619c603d393039cfa0daed99234f9bb6e09882fe8f0eae71f93e19ab333" Nov 04 03:45:03.288205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.288161 4959 scope.go:122] "RemoveContainer" containerID="c8e885a9dc372c93de5315d880cd926a0c358093cae6fc2a3b4f1624e3c0c812" Nov 04 03:45:03.364912 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.364882 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:03.600095 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.600067 4959 generic.go:356] "Generic (PLEG): container finished" podID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerID="c7ccad5346162be8963a048174742b38ddca725e317330a9de869067569239a2" exitCode=60 Nov 04 03:45:03.600502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.600145 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerDied","Data":"c7ccad5346162be8963a048174742b38ddca725e317330a9de869067569239a2"} Nov 04 03:45:03.600502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.600245 4959 scope.go:122] "RemoveContainer" containerID="c125dd2422fb1931385a9820fe53f3bab9c8dfd42809b084cbca61c752a48cc9" Nov 04 03:45:03.600662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.600523 4959 scope.go:122] "RemoveContainer" containerID="c7ccad5346162be8963a048174742b38ddca725e317330a9de869067569239a2" Nov 04 03:45:03.600829 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:03.600807 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf_odh-ai-gateway-infra(f62c0ba8-77aa-4948-a872-29baa270c56d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" Nov 04 03:45:03.602250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.602226 4959 generic.go:356] "Generic (PLEG): container finished" podID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerID="196f25cc19b127e5ce2e7bd30134b31dde4a87eefe5ed376f965ecfb6c5edd0c" exitCode=60 Nov 04 03:45:03.602336 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.602309 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" event={"ID":"ab5c9294-9c7d-42c8-9934-77a8dd8636d0","Type":"ContainerDied","Data":"196f25cc19b127e5ce2e7bd30134b31dde4a87eefe5ed376f965ecfb6c5edd0c"} Nov 04 03:45:03.602550 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.602532 4959 scope.go:122] "RemoveContainer" containerID="196f25cc19b127e5ce2e7bd30134b31dde4a87eefe5ed376f965ecfb6c5edd0c" Nov 04 03:45:03.602804 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:03.602783 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4_odh-ai-gateway-infra(ab5c9294-9c7d-42c8-9934-77a8dd8636d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" Nov 04 03:45:03.604208 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.604188 4959 generic.go:356] "Generic (PLEG): container finished" podID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerID="433836a743ab6425c5b32796ab6d47697381617bc9878e3296c20b3017abcc33" exitCode=60 Nov 04 03:45:03.604294 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.604238 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerDied","Data":"433836a743ab6425c5b32796ab6d47697381617bc9878e3296c20b3017abcc33"} Nov 04 03:45:03.604538 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.604415 4959 scope.go:122] "RemoveContainer" containerID="433836a743ab6425c5b32796ab6d47697381617bc9878e3296c20b3017abcc33" Nov 04 03:45:03.604649 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:03.604630 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2_odh-ai-gateway-infra(92bdfe44-4638-4d2c-87f4-a69f1eb22fa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" Nov 04 03:45:03.647450 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.647428 4959 scope.go:122] "RemoveContainer" containerID="3bb190ac884a8ed0851f0182c31788e8a8fac91c5ae15bab5536035015b85a7b" Nov 04 03:45:03.659464 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:03.659432 4959 scope.go:122] "RemoveContainer" containerID="0360b51df75733978d874834b116358fb91d3c4002ba559282199248d85adf4a" Nov 04 03:45:04.611578 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:04.611535 4959 scope.go:122] "RemoveContainer" containerID="433836a743ab6425c5b32796ab6d47697381617bc9878e3296c20b3017abcc33" Nov 04 03:45:04.612021 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:04.611818 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2_odh-ai-gateway-infra(92bdfe44-4638-4d2c-87f4-a69f1eb22fa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" Nov 04 03:45:04.613169 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:04.613140 4959 scope.go:122] "RemoveContainer" containerID="c7ccad5346162be8963a048174742b38ddca725e317330a9de869067569239a2" Nov 04 03:45:04.613390 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:04.613366 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf_odh-ai-gateway-infra(f62c0ba8-77aa-4948-a872-29baa270c56d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" Nov 04 03:45:04.614640 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:04.614621 4959 scope.go:122] "RemoveContainer" containerID="196f25cc19b127e5ce2e7bd30134b31dde4a87eefe5ed376f965ecfb6c5edd0c" Nov 04 03:45:04.614812 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:04.614797 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4_odh-ai-gateway-infra(ab5c9294-9c7d-42c8-9934-77a8dd8636d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" Nov 04 03:45:07.793205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:07.793169 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:08.145593 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:08.145561 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:12.966947 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:12.966911 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:13.792076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:13.792048 4959 scope.go:122] "RemoveContainer" containerID="196f25cc19b127e5ce2e7bd30134b31dde4a87eefe5ed376f965ecfb6c5edd0c" Nov 04 03:45:14.670898 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:14.670864 4959 generic.go:356] "Generic (PLEG): container finished" podID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerID="6f73e13f57853e3a3d8ebde59b392bff7ec20a4a8b97590cc6342922c633f2ba" exitCode=60 Nov 04 03:45:14.671377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:14.670939 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" event={"ID":"ab5c9294-9c7d-42c8-9934-77a8dd8636d0","Type":"ContainerDied","Data":"6f73e13f57853e3a3d8ebde59b392bff7ec20a4a8b97590cc6342922c633f2ba"} Nov 04 03:45:14.671377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:14.671005 4959 scope.go:122] "RemoveContainer" containerID="196f25cc19b127e5ce2e7bd30134b31dde4a87eefe5ed376f965ecfb6c5edd0c" Nov 04 03:45:14.671547 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:14.671408 4959 scope.go:122] "RemoveContainer" containerID="6f73e13f57853e3a3d8ebde59b392bff7ec20a4a8b97590cc6342922c633f2ba" Nov 04 03:45:14.671764 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:14.671730 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4_odh-ai-gateway-infra(ab5c9294-9c7d-42c8-9934-77a8dd8636d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" Nov 04 03:45:14.792560 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:14.792531 4959 scope.go:122] "RemoveContainer" containerID="433836a743ab6425c5b32796ab6d47697381617bc9878e3296c20b3017abcc33" Nov 04 03:45:15.683556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:15.683524 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerStarted","Data":"f9850584ca13e632248ac017be0556b7500e197d393a61b69246b41f38c57a75"} Nov 04 03:45:15.792649 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:15.792619 4959 scope.go:122] "RemoveContainer" containerID="c7ccad5346162be8963a048174742b38ddca725e317330a9de869067569239a2" Nov 04 03:45:15.799140 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:15.799114 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4"] Nov 04 03:45:15.952033 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:15.952004 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:16.074619 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.074588 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f\" (UniqueName: \"kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f\") pod \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " Nov 04 03:45:16.074816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.074666 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca\") pod \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\" (UID: \"ab5c9294-9c7d-42c8-9934-77a8dd8636d0\") " Nov 04 03:45:16.075068 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.075039 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca" pod "ab5c9294-9c7d-42c8-9934-77a8dd8636d0" (UID: "ab5c9294-9c7d-42c8-9934-77a8dd8636d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:45:16.075219 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.075139 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:16.077063 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.077047 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f" pod "ab5c9294-9c7d-42c8-9934-77a8dd8636d0" (UID: "ab5c9294-9c7d-42c8-9934-77a8dd8636d0"). InnerVolumeSpecName "kube-api-access-cpk8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:45:16.176413 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.176380 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpk8f\" (UniqueName: \"kubernetes.io/projected/ab5c9294-9c7d-42c8-9934-77a8dd8636d0-kube-api-access-cpk8f\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:16.355326 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:16.355289 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:45:16.692101 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.692016 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerStarted","Data":"ece5f9cf1aaf0846b6ad7b42df8e632f9fcd82eac179cc4eb50d2b299daf40bb"} Nov 04 03:45:16.693352 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.693328 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" Nov 04 03:45:16.693506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.693351 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4" event={"ID":"ab5c9294-9c7d-42c8-9934-77a8dd8636d0","Type":"ContainerDied","Data":"620bbec0a7a5f9e1c4fdb798e69e63570eec52a34839c214fd70ac82f9aee8d5"} Nov 04 03:45:16.693506 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.693398 4959 scope.go:122] "RemoveContainer" containerID="6f73e13f57853e3a3d8ebde59b392bff7ec20a4a8b97590cc6342922c633f2ba" Nov 04 03:45:16.695234 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.695207 4959 generic.go:356] "Generic (PLEG): container finished" podID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerID="f9850584ca13e632248ac017be0556b7500e197d393a61b69246b41f38c57a75" exitCode=60 Nov 04 03:45:16.695348 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.695279 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerDied","Data":"f9850584ca13e632248ac017be0556b7500e197d393a61b69246b41f38c57a75"} Nov 04 03:45:16.695632 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.695618 4959 scope.go:122] "RemoveContainer" containerID="f9850584ca13e632248ac017be0556b7500e197d393a61b69246b41f38c57a75" Nov 04 03:45:16.695837 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:16.695821 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2_odh-ai-gateway-infra(92bdfe44-4638-4d2c-87f4-a69f1eb22fa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" Nov 04 03:45:16.702850 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.702835 4959 scope.go:122] "RemoveContainer" containerID="433836a743ab6425c5b32796ab6d47697381617bc9878e3296c20b3017abcc33" Nov 04 03:45:16.813853 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:16.813822 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2"] Nov 04 03:45:17.036781 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.036696 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4"] Nov 04 03:45:17.043316 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.043277 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c5bec0-29896065-vg8t4"] Nov 04 03:45:17.713672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.713636 4959 generic.go:356] "Generic (PLEG): container finished" podID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerID="ece5f9cf1aaf0846b6ad7b42df8e632f9fcd82eac179cc4eb50d2b299daf40bb" exitCode=60 Nov 04 03:45:17.714175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.713714 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerDied","Data":"ece5f9cf1aaf0846b6ad7b42df8e632f9fcd82eac179cc4eb50d2b299daf40bb"} Nov 04 03:45:17.714175 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.713788 4959 scope.go:122] "RemoveContainer" containerID="c7ccad5346162be8963a048174742b38ddca725e317330a9de869067569239a2" Nov 04 03:45:17.714324 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.714196 4959 scope.go:122] "RemoveContainer" containerID="ece5f9cf1aaf0846b6ad7b42df8e632f9fcd82eac179cc4eb50d2b299daf40bb" Nov 04 03:45:17.714542 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:17.714510 4959 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf_odh-ai-gateway-infra(f62c0ba8-77aa-4948-a872-29baa270c56d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" Nov 04 03:45:17.737233 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.737203 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:17.793553 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.793520 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:17.818084 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.818055 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf"] Nov 04 03:45:17.871386 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.871357 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:17.998273 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.998186 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp\") pod \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " Nov 04 03:45:17.998273 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.998231 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca\") pod \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\" (UID: \"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7\") " Nov 04 03:45:17.998672 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:17.998645 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca" pod "92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" (UID: "92bdfe44-4638-4d2c-87f4-a69f1eb22fa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:45:18.000538 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.000448 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp" pod "92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" (UID: "92bdfe44-4638-4d2c-87f4-a69f1eb22fa7"). InnerVolumeSpecName "kube-api-access-ltdrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:45:18.099437 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.099400 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdrp\" (UniqueName: \"kubernetes.io/projected/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-kube-api-access-ltdrp\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:18.099437 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.099430 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:18.725826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.725803 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" Nov 04 03:45:18.725826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.725814 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2" event={"ID":"92bdfe44-4638-4d2c-87f4-a69f1eb22fa7","Type":"ContainerDied","Data":"e20550ac5629d08301d351ac5ca420523fbe148af622bbce31a2be716e676083"} Nov 04 03:45:18.726303 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.725867 4959 scope.go:122] "RemoveContainer" containerID="f9850584ca13e632248ac017be0556b7500e197d393a61b69246b41f38c57a75" Nov 04 03:45:18.802283 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.801300 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" path="/var/lib/kubelet/pods/ab5c9294-9c7d-42c8-9934-77a8dd8636d0/volumes" Nov 04 03:45:18.873840 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.873814 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:18.953744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.953706 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2"] Nov 04 03:45:18.959846 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:18.959818 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-13190c-29896065-gkdc2"] Nov 04 03:45:19.010043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.009954 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq\" (UniqueName: \"kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq\") pod \"f62c0ba8-77aa-4948-a872-29baa270c56d\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " Nov 04 03:45:19.010216 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.010095 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca\") pod \"f62c0ba8-77aa-4948-a872-29baa270c56d\" (UID: \"f62c0ba8-77aa-4948-a872-29baa270c56d\") " Nov 04 03:45:19.010573 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.010548 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca" pod "f62c0ba8-77aa-4948-a872-29baa270c56d" (UID: "f62c0ba8-77aa-4948-a872-29baa270c56d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:45:19.012339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.012316 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq" pod "f62c0ba8-77aa-4948-a872-29baa270c56d" (UID: "f62c0ba8-77aa-4948-a872-29baa270c56d"). InnerVolumeSpecName "kube-api-access-cdbsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:45:19.111810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.111775 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f62c0ba8-77aa-4948-a872-29baa270c56d-maas-api-service-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:19.111810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.111810 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdbsq\" (UniqueName: \"kubernetes.io/projected/f62c0ba8-77aa-4948-a872-29baa270c56d-kube-api-access-cdbsq\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:45:19.733898 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.733859 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" event={"ID":"f62c0ba8-77aa-4948-a872-29baa270c56d","Type":"ContainerDied","Data":"3f8b81b76d22bc94e554c378ced4dcdbd5e20c9a9bec3d5729a749221f6039fd"} Nov 04 03:45:19.734390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.733920 4959 scope.go:122] "RemoveContainer" containerID="ece5f9cf1aaf0846b6ad7b42df8e632f9fcd82eac179cc4eb50d2b299daf40bb" Nov 04 03:45:19.734390 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.733874 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf" Nov 04 03:45:19.907234 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.907199 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf"] Nov 04 03:45:19.957595 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:19.957559 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5d459a-29896065-fs7pf"] Nov 04 03:45:20.799362 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:20.799330 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" path="/var/lib/kubelet/pods/92bdfe44-4638-4d2c-87f4-a69f1eb22fa7/volumes" Nov 04 03:45:20.799800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:20.799733 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" path="/var/lib/kubelet/pods/f62c0ba8-77aa-4948-a872-29baa270c56d/volumes" Nov 04 03:45:22.546076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:22.546042 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:27.335601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:27.335565 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:27.793180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:27.793094 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:32.236800 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:32.236746 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:37.057228 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:37.057184 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:37.793413 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:37.793371 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:41.838782 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:41.838750 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:46.354162 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:45:46.354120 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:45:46.639591 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:46.639513 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:47.793193 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:47.793157 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:51.457050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:51.457013 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:53.047237 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:53.047203 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk"] Nov 04 03:45:53.097105 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:53.097069 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de351b77-bhfvk"] Nov 04 03:45:54.798313 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:54.798280 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c0c26a-10e7-45fb-b097-a0af8b83b1dd" path="/var/lib/kubelet/pods/26c0c26a-10e7-45fb-b097-a0af8b83b1dd/volumes" Nov 04 03:45:56.238439 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:56.238404 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:45:57.047260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:57.047221 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq"] Nov 04 03:45:57.097597 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:57.097561 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f975cb10-8jsxq"] Nov 04 03:45:57.793231 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:57.793200 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:45:58.798620 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:45:58.798589 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30347b50-be86-43da-b8c2-fb6ae6c872a9" path="/var/lib/kubelet/pods/30347b50-be86-43da-b8c2-fb6ae6c872a9/volumes" Nov 04 03:46:01.034185 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:01.034147 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:02.584493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:02.584438 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:03.333905 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:03.333870 4959 scope.go:122] "RemoveContainer" containerID="55a48c670f90ec323e5ca933e057d57386eff822181d74f53bfa260da6732a9b" Nov 04 03:46:05.837735 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:05.837704 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:07.046826 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:07.046784 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t"] Nov 04 03:46:07.097227 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:07.097198 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e8a8d38-5db7t"] Nov 04 03:46:07.793415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:07.793376 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:46:08.400653 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.400620 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn"] Nov 04 03:46:08.401058 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401044 4959 cpu_manager.go:414] "containerMap: removing container" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401059 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401079 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:08.401104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401085 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:08.401104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401095 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:08.401104 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401100 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401109 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401114 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401123 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401127 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401136 4959 cpu_manager.go:414] "containerMap: removing container" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401141 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401155 4959 cpu_manager.go:414] "containerMap: removing container" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401161 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401167 4959 cpu_manager.go:414] "containerMap: removing container" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401172 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401248 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401253 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401256 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:08.401637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401263 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:08.401637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401268 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401274 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401280 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.401637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401285 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5c9294-9c7d-42c8-9934-77a8dd8636d0" containerName="cleanup" Nov 04 03:46:08.401637 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.401290 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="92bdfe44-4638-4d2c-87f4-a69f1eb22fa7" containerName="cleanup" Nov 04 03:46:08.405694 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.405679 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.512704 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.512669 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:46:08.512704 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.512703 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:46:08.512888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.512679 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:46:08.512888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.512679 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:46:08.514823 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.514803 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn"] Nov 04 03:46:08.524548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.524522 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.524762 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.524733 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.524874 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.524766 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlpj\" (UniqueName: \"kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.625906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.625878 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.625906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.625912 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.626096 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.625929 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlpj\" (UniqueName: \"kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.626537 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.626515 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.628380 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.628360 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.687748 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.687689 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlpj\" (UniqueName: \"kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj\") pod \"maas-api-revoke-sub-35d81421-sg6vn\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.717644 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.717620 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:08.806662 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:08.806620 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748f68c1-3683-4285-880e-80f796f71dcf" path="/var/lib/kubelet/pods/748f68c1-3683-4285-880e-80f796f71dcf/volumes" Nov 04 03:46:09.048771 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:09.048683 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9"] Nov 04 03:46:09.052507 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:09.052465 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j"] Nov 04 03:46:09.054220 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:09.054203 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c817be0-tqtj9"] Nov 04 03:46:09.099013 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:09.098988 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83cf9538-knh9j"] Nov 04 03:46:09.100742 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:09.100719 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn"] Nov 04 03:46:09.132448 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:46:09.132419 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6e51a3_6144_4a00_be33_eb9c2272e81a.slice/crio-1057917c30e236eb0ec18163ad51a218e2079f9a81e9aefd9206718090e088b3 WatchSource:0}: Error finding container 1057917c30e236eb0ec18163ad51a218e2079f9a81e9aefd9206718090e088b3: Status 404 returned error can't find the container with id 1057917c30e236eb0ec18163ad51a218e2079f9a81e9aefd9206718090e088b3 Nov 04 03:46:10.003355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:10.003321 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" event={"ID":"1e6e51a3-6144-4a00-be33-eb9c2272e81a","Type":"ContainerStarted","Data":"911d28aef34f8993fd16c3c86da6331c9d6ec434fb9a92e14907a843b2d454df"} Nov 04 03:46:10.003801 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:10.003383 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" event={"ID":"1e6e51a3-6144-4a00-be33-eb9c2272e81a","Type":"ContainerStarted","Data":"1057917c30e236eb0ec18163ad51a218e2079f9a81e9aefd9206718090e088b3"} Nov 04 03:46:10.122376 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:10.122316 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" podStartSLOduration=2.122301302 podStartE2EDuration="2.122301302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:46:10.119734374 +0000 UTC m=+1747.503162414" watchObservedRunningTime="2026-11-04 03:46:10.122301302 +0000 UTC m=+1747.505729341" Nov 04 03:46:10.737756 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:10.737722 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:10.798391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:10.798356 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c46eb4-6a05-441e-a802-53a6b59219bd" path="/var/lib/kubelet/pods/01c46eb4-6a05-441e-a802-53a6b59219bd/volumes" Nov 04 03:46:10.798804 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:10.798789 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99" path="/var/lib/kubelet/pods/bb5be1d9-bb1e-4688-9b5a-d9b0f8494b99/volumes" Nov 04 03:46:11.010099 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:11.010010 4959 generic.go:356] "Generic (PLEG): container finished" podID="1e6e51a3-6144-4a00-be33-eb9c2272e81a" containerID="911d28aef34f8993fd16c3c86da6331c9d6ec434fb9a92e14907a843b2d454df" exitCode=0 Nov 04 03:46:11.010454 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:11.010100 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" event={"ID":"1e6e51a3-6144-4a00-be33-eb9c2272e81a","Type":"ContainerDied","Data":"911d28aef34f8993fd16c3c86da6331c9d6ec434fb9a92e14907a843b2d454df"} Nov 04 03:46:12.153335 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.153312 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:12.264924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.264893 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca\") pod \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " Nov 04 03:46:12.265075 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.265026 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj\" (UniqueName: \"kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj\") pod \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " Nov 04 03:46:12.265075 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.265062 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca\") pod \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\" (UID: \"1e6e51a3-6144-4a00-be33-eb9c2272e81a\") " Nov 04 03:46:12.265431 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.265401 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca" pod "1e6e51a3-6144-4a00-be33-eb9c2272e81a" (UID: "1e6e51a3-6144-4a00-be33-eb9c2272e81a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:46:12.267212 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.267166 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj" pod "1e6e51a3-6144-4a00-be33-eb9c2272e81a" (UID: "1e6e51a3-6144-4a00-be33-eb9c2272e81a"). InnerVolumeSpecName "kube-api-access-4wlpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:46:12.267366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.267351 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca" pod "1e6e51a3-6144-4a00-be33-eb9c2272e81a" (UID: "1e6e51a3-6144-4a00-be33-eb9c2272e81a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:46:12.366715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.366683 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:12.366715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.366710 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wlpj\" (UniqueName: \"kubernetes.io/projected/1e6e51a3-6144-4a00-be33-eb9c2272e81a-kube-api-access-4wlpj\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:12.366715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:12.366720 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e6e51a3-6144-4a00-be33-eb9c2272e81a-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:13.021201 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:13.021175 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" Nov 04 03:46:13.021412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:13.021214 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35d81421-sg6vn" event={"ID":"1e6e51a3-6144-4a00-be33-eb9c2272e81a","Type":"ContainerDied","Data":"1057917c30e236eb0ec18163ad51a218e2079f9a81e9aefd9206718090e088b3"} Nov 04 03:46:13.021412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:13.021260 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1057917c30e236eb0ec18163ad51a218e2079f9a81e9aefd9206718090e088b3" Nov 04 03:46:13.779340 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:46:13.779302 4959 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2c682677175: secret "agc-cred-f2c682677175" not found Nov 04 03:46:13.779340 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:46:13.779333 4959 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-cd8696ff5-4976v: secret "agc-cred-f2c682677175" not found Nov 04 03:46:13.779908 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:46:13.779396 4959 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials podName:bbf6e2e1-0717-43a8-919a-46f87ff44be0 nodeName:}" failed. No retries permitted until 2026-11-04 03:48:15.779378176 +0000 UTC m=+1873.162806210 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials") pod "payload-processing-external-model-cd8696ff5-4976v" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0") : secret "agc-cred-f2c682677175" not found Nov 04 03:46:15.541013 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:15.540975 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:16.354360 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:46:16.354319 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:46:17.793715 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:17.793677 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:46:20.333602 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:20.333567 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:21.701681 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:21.701644 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:21.929147 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:21.929115 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd8696ff5-4976v"] Nov 04 03:46:21.958256 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:21.958167 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:21.979435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:21.979408 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf9557897-mngcx"] Nov 04 03:46:21.980729 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:21.980707 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:46:22.212030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:22.211946 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:23.046304 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:23.046272 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm"] Nov 04 03:46:23.096635 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:23.096607 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4485f952-9mknm"] Nov 04 03:46:23.970712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:23.970681 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:24.798355 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:24.798325 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcdb0a59-bf2a-4f15-affc-d11fd286af80" path="/var/lib/kubelet/pods/bcdb0a59-bf2a-4f15-affc-d11fd286af80/volumes" Nov 04 03:46:25.045963 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:25.045927 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb"] Nov 04 03:46:25.096403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:25.096370 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-434a96c9-b8rnb"] Nov 04 03:46:25.243695 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:25.243651 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:26.453809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:26.453774 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:46:26.800754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:26.800673 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf" path="/var/lib/kubelet/pods/9164eb2d-bc55-4b04-92d2-5fb3b95dcfaf/volumes" Nov 04 03:46:26.929396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:26.929357 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" podUID="bbf6e2e1-0717-43a8-919a-46f87ff44be0" containerName="payload-processing" containerID="cri-o://37b1895fb29845120b4b9a3184aee03e25cc67fb6da720931ea7f5a375a4b6aa" gracePeriod=25 Nov 04 03:46:26.980058 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:26.980007 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" containerID="cri-o://8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b" gracePeriod=25 Nov 04 03:46:27.100006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.099967 4959 generic.go:356] "Generic (PLEG): container finished" podID="bbf6e2e1-0717-43a8-919a-46f87ff44be0" containerID="37b1895fb29845120b4b9a3184aee03e25cc67fb6da720931ea7f5a375a4b6aa" exitCode=0 Nov 04 03:46:27.100171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.100140 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" event={"ID":"bbf6e2e1-0717-43a8-919a-46f87ff44be0","Type":"ContainerDied","Data":"37b1895fb29845120b4b9a3184aee03e25cc67fb6da720931ea7f5a375a4b6aa"} Nov 04 03:46:27.302721 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.302688 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:46:27.355420 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.355394 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:46:27.431050 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.431012 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume\") pod \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " Nov 04 03:46:27.431214 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.431128 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials\") pod \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " Nov 04 03:46:27.431214 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.431167 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg\") pod \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " Nov 04 03:46:27.431319 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.431239 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume\") pod \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\" (UID: \"bbf6e2e1-0717-43a8-919a-46f87ff44be0\") " Nov 04 03:46:27.432518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.432487 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume" pod "bbf6e2e1-0717-43a8-919a-46f87ff44be0" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:46:27.432916 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.432892 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume" pod "bbf6e2e1-0717-43a8-919a-46f87ff44be0" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:46:27.435746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.435717 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials" pod "bbf6e2e1-0717-43a8-919a-46f87ff44be0" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:46:27.437174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.437138 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg" pod "bbf6e2e1-0717-43a8-919a-46f87ff44be0" (UID: "bbf6e2e1-0717-43a8-919a-46f87ff44be0"). InnerVolumeSpecName "kube-api-access-428wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:46:27.532011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.531974 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume\") pod \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " Nov 04 03:46:27.532548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532073 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials\") pod \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " Nov 04 03:46:27.532548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532128 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc\") pod \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " Nov 04 03:46:27.532548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532181 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume\") pod \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\" (UID: \"a65c2603-4b46-4dc2-8f53-02985e3a5d5c\") " Nov 04 03:46:27.532747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532553 4959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-routing-overlay-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.532747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532574 4959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-provider-credentials\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.532747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532589 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428wg\" (UniqueName: \"kubernetes.io/projected/bbf6e2e1-0717-43a8-919a-46f87ff44be0-kube-api-access-428wg\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.532747 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.532604 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf6e2e1-0717-43a8-919a-46f87ff44be0-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.533309 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.533003 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume" pod "a65c2603-4b46-4dc2-8f53-02985e3a5d5c" (UID: "a65c2603-4b46-4dc2-8f53-02985e3a5d5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:46:27.533426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.533362 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume" pod "a65c2603-4b46-4dc2-8f53-02985e3a5d5c" (UID: "a65c2603-4b46-4dc2-8f53-02985e3a5d5c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:46:27.535859 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.535825 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials" pod "a65c2603-4b46-4dc2-8f53-02985e3a5d5c" (UID: "a65c2603-4b46-4dc2-8f53-02985e3a5d5c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:46:27.535974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.535903 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc" pod "a65c2603-4b46-4dc2-8f53-02985e3a5d5c" (UID: "a65c2603-4b46-4dc2-8f53-02985e3a5d5c"). InnerVolumeSpecName "kube-api-access-qtknc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:46:27.634205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.634097 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtknc\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-kube-api-access-qtknc\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.634205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.634142 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.634205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.634159 4959 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-routing-overlay-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:27.634205 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:27.634173 4959 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65c2603-4b46-4dc2-8f53-02985e3a5d5c-provider-credentials\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:46:28.111518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.111112 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" event={"ID":"bbf6e2e1-0717-43a8-919a-46f87ff44be0","Type":"ContainerDied","Data":"43d362eddfe7415a37c41852ef478cc8492a8379562e2828c0874a9674339225"} Nov 04 03:46:28.111518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.111187 4959 scope.go:122] "RemoveContainer" containerID="37b1895fb29845120b4b9a3184aee03e25cc67fb6da720931ea7f5a375a4b6aa" Nov 04 03:46:28.111518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.111334 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-cd8696ff5-4976v" Nov 04 03:46:28.123557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.121328 4959 generic.go:356] "Generic (PLEG): container finished" podID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerID="8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b" exitCode=0 Nov 04 03:46:28.123557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.121439 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" event={"ID":"a65c2603-4b46-4dc2-8f53-02985e3a5d5c","Type":"ContainerDied","Data":"8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b"} Nov 04 03:46:28.123557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.121636 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" event={"ID":"a65c2603-4b46-4dc2-8f53-02985e3a5d5c","Type":"ContainerDied","Data":"c92f026119ff7b9730f196054dbf8b6580b0bdb6832f5271da5c3232b4d0a8e3"} Nov 04 03:46:28.123557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.121728 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf9557897-mngcx" Nov 04 03:46:28.148640 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.148619 4959 scope.go:122] "RemoveContainer" containerID="8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b" Nov 04 03:46:28.159978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.159960 4959 scope.go:122] "RemoveContainer" containerID="8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b" Nov 04 03:46:28.160456 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:46:28.160434 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b\": container with ID starting with 8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b not found: ID does not exist" containerID="8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b" Nov 04 03:46:28.160559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.160484 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b"} err="failed to get container status \"8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b\": rpc error: code = NotFound desc = could not find container \"8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b\": container with ID starting with 8813d22cecca0e836b4c6883f708a7838bf98a00212bfe7ba6561dd3b614b47b not found: ID does not exist" Nov 04 03:46:28.295810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.295710 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf9557897-mngcx"] Nov 04 03:46:28.355767 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.355732 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf9557897-mngcx"] Nov 04 03:46:28.443548 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.443517 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd8696ff5-4976v"] Nov 04 03:46:28.445914 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.445885 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-cd8696ff5-4976v"] Nov 04 03:46:28.807494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.803921 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" path="/var/lib/kubelet/pods/a65c2603-4b46-4dc2-8f53-02985e3a5d5c/volumes" Nov 04 03:46:28.807494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:28.804486 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf6e2e1-0717-43a8-919a-46f87ff44be0" path="/var/lib/kubelet/pods/bbf6e2e1-0717-43a8-919a-46f87ff44be0/volumes" Nov 04 03:46:30.353423 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:30.353388 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:30.657535 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:30.657438 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:31.503659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:31.503622 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:31.761689 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:31.761608 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:32.661447 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:32.661411 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:35.640636 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:35.640601 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:37.238123 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:37.238081 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:39.853170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:39.853133 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:41.648102 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:41.648067 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:43.122391 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:43.122358 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:45.027347 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:45.027314 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:46.354584 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:46:46.354547 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:46:47.024957 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:47.024920 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:49.787183 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:49.787141 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:50.042060 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:50.041962 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:51.848631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:51.848592 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:54.565904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:54.565870 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:46:55.165237 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:55.165191 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q"] Nov 04 03:46:55.165516 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:55.165452 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" containerID="cri-o://cc77707d821e4c2202bd39199796a40701c1929ef6adbc24fbf9eae900bb786b" gracePeriod=30 Nov 04 03:46:55.277965 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:55.277933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:56.110260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.110221 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:56.631119 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.631088 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:46:56.950489 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950404 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk"] Nov 04 03:46:56.950831 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950817 4959 cpu_manager.go:414] "containerMap: removing container" podUID="bbf6e2e1-0717-43a8-919a-46f87ff44be0" containerName="payload-processing" Nov 04 03:46:56.950878 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950832 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf6e2e1-0717-43a8-919a-46f87ff44be0" containerName="payload-processing" Nov 04 03:46:56.950878 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950845 4959 cpu_manager.go:414] "containerMap: removing container" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" Nov 04 03:46:56.950878 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950850 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" Nov 04 03:46:56.950878 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950877 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950882 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950890 4959 cpu_manager.go:414] "containerMap: removing container" podUID="1e6e51a3-6144-4a00-be33-eb9c2272e81a" containerName="revoke-keys" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950895 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6e51a3-6144-4a00-be33-eb9c2272e81a" containerName="revoke-keys" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950956 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62c0ba8-77aa-4948-a872-29baa270c56d" containerName="cleanup" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950963 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6e51a3-6144-4a00-be33-eb9c2272e81a" containerName="revoke-keys" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950970 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf6e2e1-0717-43a8-919a-46f87ff44be0" containerName="payload-processing" Nov 04 03:46:56.951024 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.950978 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65c2603-4b46-4dc2-8f53-02985e3a5d5c" containerName="payload-processing" Nov 04 03:46:56.954114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:56.954099 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.034812 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.034780 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.034924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.034822 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlwc\" (UniqueName: \"kubernetes.io/projected/c1ec031a-04b7-4479-a109-0c1749bc2fe9-kube-api-access-vjlwc\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.034924 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.034842 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.035014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.034918 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.035014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.034961 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.035014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.034983 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.035164 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.035013 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.035164 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.035055 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.035235 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.035186 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.136567 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136530 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136600 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136631 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlwc\" (UniqueName: \"kubernetes.io/projected/c1ec031a-04b7-4479-a109-0c1749bc2fe9-kube-api-access-vjlwc\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136650 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136670 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136690 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136710 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136746 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.136772 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.137000 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.137048 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.137195 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.137380 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.137581 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.137564 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.139264 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.139242 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.139396 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.139380 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.162910 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.162887 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk"] Nov 04 03:46:57.198837 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.198807 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1ec031a-04b7-4479-a109-0c1749bc2fe9-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.198950 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.198901 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlwc\" (UniqueName: \"kubernetes.io/projected/c1ec031a-04b7-4479-a109-0c1749bc2fe9-kube-api-access-vjlwc\") pod \"maas-default-gateway-openshift-default-6dcf757899-hx8xk\" (UID: \"c1ec031a-04b7-4479-a109-0c1749bc2fe9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.267258 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.267199 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:57.692867 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.692831 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk"] Nov 04 03:46:57.730331 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:46:57.730304 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ec031a_04b7_4479_a109_0c1749bc2fe9.slice/crio-eb2dbff8eef2ab985211fce895403796ffa974620ed36dbaa5b5745263de08d7 WatchSource:0}: Error finding container eb2dbff8eef2ab985211fce895403796ffa974620ed36dbaa5b5745263de08d7: Status 404 returned error can't find the container with id eb2dbff8eef2ab985211fce895403796ffa974620ed36dbaa5b5745263de08d7 Nov 04 03:46:57.732643 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.732614 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:46:57.732725 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.732678 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:46:57.732725 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:57.732709 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:46:58.308426 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:58.308390 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" event={"ID":"c1ec031a-04b7-4479-a109-0c1749bc2fe9","Type":"ContainerStarted","Data":"1599892c3686fe207b3ca44b3ad84eb2a2a097e6d4cc6ccf58222e9db65a07cc"} Nov 04 03:46:58.308905 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:58.308444 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" event={"ID":"c1ec031a-04b7-4479-a109-0c1749bc2fe9","Type":"ContainerStarted","Data":"eb2dbff8eef2ab985211fce895403796ffa974620ed36dbaa5b5745263de08d7"} Nov 04 03:46:59.267968 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:59.267936 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:46:59.269768 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:59.269737 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:46:59.269906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:59.269794 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:46:59.270705 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:59.270681 4959 patch_prober.go:29] interesting pod/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 04 03:46:59.270810 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:46:59.270723 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 04 03:47:00.267612 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.267574 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:47:00.267982 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.267661 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:47:00.335894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.335862 4959 generic.go:356] "Generic (PLEG): container finished" podID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerID="cc77707d821e4c2202bd39199796a40701c1929ef6adbc24fbf9eae900bb786b" exitCode=0 Nov 04 03:47:00.336061 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.335947 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" event={"ID":"655034e2-d0d9-475d-b1b5-e5a412eb9bc5","Type":"ContainerDied","Data":"cc77707d821e4c2202bd39199796a40701c1929ef6adbc24fbf9eae900bb786b"} Nov 04 03:47:00.336061 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.336023 4959 scope.go:122] "RemoveContainer" containerID="2611813c2acbae393b0b64416cf8e807dc41775ee540d47a63adb5e7b973d584" Nov 04 03:47:00.466339 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.466315 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:47:00.570993 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.570959 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571189 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571026 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571189 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571072 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571189 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571100 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571189 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571167 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571239 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571276 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571296 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:00.571368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571308 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571381 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:00.571596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571390 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token\") pod \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\" (UID: \"655034e2-d0d9-475d-b1b5-e5a412eb9bc5\") " Nov 04 03:47:00.571596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571531 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:00.571809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571783 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:00.571877 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.571787 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:47:00.572120 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.572099 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.572180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.572127 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.572180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.572145 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.572180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.572159 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.572180 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.572175 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.573659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.573636 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "kube-api-access-lbwtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:00.573659 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.573646 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:00.573848 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.573830 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:00.573896 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.573859 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo" pod "655034e2-d0d9-475d-b1b5-e5a412eb9bc5" (UID: "655034e2-d0d9-475d-b1b5-e5a412eb9bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:47:00.639894 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.639840 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podStartSLOduration=4.639826049 podStartE2EDuration="4.639826049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:46:58.498500628 +0000 UTC m=+1795.881928668" watchObservedRunningTime="2026-11-04 03:47:00.639826049 +0000 UTC m=+1798.023254088" Nov 04 03:47:00.672935 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.672900 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.672935 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.672924 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwtp\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-kube-api-access-lbwtp\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.672935 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.672934 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:00.672935 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:00.672943 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655034e2-d0d9-475d-b1b5-e5a412eb9bc5-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:01.267586 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.267547 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:47:01.267799 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.267619 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:47:01.342614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.342575 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" event={"ID":"655034e2-d0d9-475d-b1b5-e5a412eb9bc5","Type":"ContainerDied","Data":"f6688af9dc6edd3326a5a8c9a4c2b11dc5543bc8baf6e4ac5649cb588a4be8e1"} Nov 04 03:47:01.342614 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.342604 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q" Nov 04 03:47:01.342883 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.342650 4959 scope.go:122] "RemoveContainer" containerID="cc77707d821e4c2202bd39199796a40701c1929ef6adbc24fbf9eae900bb786b" Nov 04 03:47:01.443904 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.443878 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:01.557794 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.557711 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q"] Nov 04 03:47:01.608485 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:01.608434 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-13190c-openshift-default-5b8c6555c-7l72q"] Nov 04 03:47:02.268456 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.268422 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:47:02.268876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.268504 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:47:02.375745 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.375708 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:02.810949 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.810914 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" path="/var/lib/kubelet/pods/655034e2-d0d9-475d-b1b5-e5a412eb9bc5/volumes" Nov 04 03:47:02.939271 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.939233 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:02.971351 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.971325 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:47:02.977973 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:02.977954 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:47:03.010981 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.010954 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-v5cxt_0ed988d6-f52e-4030-8805-1c214876ee88/console-operator/2.log" Nov 04 03:47:03.018816 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.018794 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-54nv5_c66837ee-2658-4ef6-bee5-d15bbe58ebde/ovn-acl-logging/0.log" Nov 04 03:47:03.214514 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.214482 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:03.275493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.268827 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:47:03.275493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.268885 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:47:03.366174 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.366146 4959 scope.go:122] "RemoveContainer" containerID="c3042c1c2f7aae7db1685129e432cd55e5ef00dd1598d4da4babe8a42123e112" Nov 04 03:47:03.374851 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.374834 4959 scope.go:122] "RemoveContainer" containerID="9fd71ef4161949e6b0f14a13050803ea13e720b4a5b5664f01e27fd205b7cd2d" Nov 04 03:47:03.470019 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:03.469956 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:04.267686 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:04.267650 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:47:04.267856 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:04.267712 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:47:05.268412 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:05.268377 4959 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hx8xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 04 03:47:05.268809 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:05.268432 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" podUID="c1ec031a-04b7-4479-a109-0c1749bc2fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 04 03:47:06.374114 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:06.374083 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:47:06.374529 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:06.374502 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:47:06.375186 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:06.375160 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hx8xk" Nov 04 03:47:06.515546 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:06.515507 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:08.299890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:08.299849 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:11.328273 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:11.328239 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:15.564621 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:15.564586 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:16.354910 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:47:16.354875 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:47:17.326156 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:17.326117 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:20.954065 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:20.954032 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:47:22.173314 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:22.173276 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:23.397621 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:23.397584 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:23.645933 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:23.645893 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:24.977716 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:24.977674 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:25.370883 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:25.370849 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p"] Nov 04 03:47:25.371073 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:25.371053 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" containerID="cri-o://b709a3054f84ced36c6720f074147e22213d6750d3d761dfd6b94d8bfa1e8890" gracePeriod=30 Nov 04 03:47:25.497289 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:25.497219 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:25.967921 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:25.967882 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:26.469403 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469370 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c"] Nov 04 03:47:26.469811 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469796 4959 cpu_manager.go:414] "containerMap: removing container" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" Nov 04 03:47:26.469854 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469812 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" Nov 04 03:47:26.469854 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469835 4959 cpu_manager.go:414] "containerMap: removing container" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" Nov 04 03:47:26.469854 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469840 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" Nov 04 03:47:26.469954 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469904 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" Nov 04 03:47:26.469954 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.469915 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="655034e2-d0d9-475d-b1b5-e5a412eb9bc5" containerName="istio-proxy" Nov 04 03:47:26.472977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.472962 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.577364 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.577333 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:47:26.577555 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.577385 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:47:26.577555 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.577401 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:47:26.577555 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.577413 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:47:26.579230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.579209 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c"] Nov 04 03:47:26.584031 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.584011 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.584106 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.584085 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.584155 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.584140 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwtjv\" (UniqueName: \"kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.685682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.685648 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.685860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.685747 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.685860 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.685778 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwtjv\" (UniqueName: \"kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.686274 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.686251 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.688250 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.688225 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.747786 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.747715 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwtjv\" (UniqueName: \"kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv\") pod \"maas-api-revoke-sub-7442a4c3-fn55c\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:26.786107 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:26.786079 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:27.156231 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:27.156206 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c"] Nov 04 03:47:27.158453 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:47:27.158428 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ced95e_bb4d_4778_a495_d59783a05932.slice/crio-ddcf04ca947c177cda875f2d01719ea1df51aeb546e3aaa3f3fff2f9af9c3420 WatchSource:0}: Error finding container ddcf04ca947c177cda875f2d01719ea1df51aeb546e3aaa3f3fff2f9af9c3420: Status 404 returned error can't find the container with id ddcf04ca947c177cda875f2d01719ea1df51aeb546e3aaa3f3fff2f9af9c3420 Nov 04 03:47:27.484305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:27.484270 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" event={"ID":"e9ced95e-bb4d-4778-a495-d59783a05932","Type":"ContainerStarted","Data":"ddcf04ca947c177cda875f2d01719ea1df51aeb546e3aaa3f3fff2f9af9c3420"} Nov 04 03:47:27.575526 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:27.575494 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:28.491755 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:28.491715 4959 generic.go:356] "Generic (PLEG): container finished" podID="e9ced95e-bb4d-4778-a495-d59783a05932" containerID="758bbab04d92260c5b4f0ccf11c3ebc85e1cfe5a6bcf11b597de99bf449c0f57" exitCode=0 Nov 04 03:47:28.492203 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:28.491801 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" event={"ID":"e9ced95e-bb4d-4778-a495-d59783a05932","Type":"ContainerDied","Data":"758bbab04d92260c5b4f0ccf11c3ebc85e1cfe5a6bcf11b597de99bf449c0f57"} Nov 04 03:47:29.630351 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.630331 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:29.716592 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.716567 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv\" (UniqueName: \"kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv\") pod \"e9ced95e-bb4d-4778-a495-d59783a05932\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " Nov 04 03:47:29.716761 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.716602 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca\") pod \"e9ced95e-bb4d-4778-a495-d59783a05932\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " Nov 04 03:47:29.716761 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.716692 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca\") pod \"e9ced95e-bb4d-4778-a495-d59783a05932\" (UID: \"e9ced95e-bb4d-4778-a495-d59783a05932\") " Nov 04 03:47:29.717043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.717011 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca" pod "e9ced95e-bb4d-4778-a495-d59783a05932" (UID: "e9ced95e-bb4d-4778-a495-d59783a05932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:47:29.718923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.718900 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca" pod "e9ced95e-bb4d-4778-a495-d59783a05932" (UID: "e9ced95e-bb4d-4778-a495-d59783a05932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:47:29.718923 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.718905 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv" pod "e9ced95e-bb4d-4778-a495-d59783a05932" (UID: "e9ced95e-bb4d-4778-a495-d59783a05932"). InnerVolumeSpecName "kube-api-access-lwtjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:29.818096 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.818016 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:29.818096 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.818044 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwtjv\" (UniqueName: \"kubernetes.io/projected/e9ced95e-bb4d-4778-a495-d59783a05932-kube-api-access-lwtjv\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:29.818096 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:29.818057 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ced95e-bb4d-4778-a495-d59783a05932-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.504484 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.504442 4959 generic.go:356] "Generic (PLEG): container finished" podID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerID="b709a3054f84ced36c6720f074147e22213d6750d3d761dfd6b94d8bfa1e8890" exitCode=0 Nov 04 03:47:30.504629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.504522 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" event={"ID":"6df7a391-1ace-4782-8f2c-410d14c4429e","Type":"ContainerDied","Data":"b709a3054f84ced36c6720f074147e22213d6750d3d761dfd6b94d8bfa1e8890"} Nov 04 03:47:30.504629 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.504588 4959 scope.go:122] "RemoveContainer" containerID="6b9e28695899ba0892316b1aba344e5b3e429e0f1bd26ae5d6cb6cc6ec282fb6" Nov 04 03:47:30.506248 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.506222 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" event={"ID":"e9ced95e-bb4d-4778-a495-d59783a05932","Type":"ContainerDied","Data":"ddcf04ca947c177cda875f2d01719ea1df51aeb546e3aaa3f3fff2f9af9c3420"} Nov 04 03:47:30.506365 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.506268 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddcf04ca947c177cda875f2d01719ea1df51aeb546e3aaa3f3fff2f9af9c3420" Nov 04 03:47:30.506365 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.506231 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7442a4c3-fn55c" Nov 04 03:47:30.604937 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.604914 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726495 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726609 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726668 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726703 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726742 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726766 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726826 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726895 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727034 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.726926 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token\") pod \"6df7a391-1ace-4782-8f2c-410d14c4429e\" (UID: \"6df7a391-1ace-4782-8f2c-410d14c4429e\") " Nov 04 03:47:30.727966 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.727647 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:30.727966 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.727772 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:47:30.727966 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.727865 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:30.727966 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.727885 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:30.728357 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.728331 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:30.731076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.731046 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "kube-api-access-lm587". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:30.731318 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.731292 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:47:30.731888 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.731863 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:47:30.732209 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.732128 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token" pod "6df7a391-1ace-4782-8f2c-410d14c4429e" (UID: "6df7a391-1ace-4782-8f2c-410d14c4429e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:30.828028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828001 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828027 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828040 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828050 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828058 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828066 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df7a391-1ace-4782-8f2c-410d14c4429e-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828075 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm587\" (UniqueName: \"kubernetes.io/projected/6df7a391-1ace-4782-8f2c-410d14c4429e-kube-api-access-lm587\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828083 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:30.828179 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:30.828090 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df7a391-1ace-4782-8f2c-410d14c4429e-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:31.079595 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:31.079557 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:31.512322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:31.512235 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" event={"ID":"6df7a391-1ace-4782-8f2c-410d14c4429e","Type":"ContainerDied","Data":"69104b54f1cc3d39fe630b73fa48a6b78ffe083351d4f48812e6ff71cdad3d3f"} Nov 04 03:47:31.512322 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:31.512282 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p" Nov 04 03:47:31.512574 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:31.512339 4959 scope.go:122] "RemoveContainer" containerID="b709a3054f84ced36c6720f074147e22213d6750d3d761dfd6b94d8bfa1e8890" Nov 04 03:47:31.744127 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:31.744093 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p"] Nov 04 03:47:31.793685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:31.793617 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5d459a-openshift-default-6c9dd9f4bf-kdb9p"] Nov 04 03:47:32.721015 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:32.720984 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:32.798513 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:32.798458 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" path="/var/lib/kubelet/pods/6df7a391-1ace-4782-8f2c-410d14c4429e/volumes" Nov 04 03:47:33.064520 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:33.064413 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:36.038393 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:36.038353 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:47:37.578975 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:37.578941 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:38.410528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.410495 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw"] Nov 04 03:47:38.518701 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.518665 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk"] Nov 04 03:47:38.867009 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.866967 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:38.877011 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.876985 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5"] Nov 04 03:47:38.877385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877374 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" Nov 04 03:47:38.877424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877387 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" Nov 04 03:47:38.877424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877400 4959 cpu_manager.go:414] "containerMap: removing container" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" Nov 04 03:47:38.877424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877404 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" Nov 04 03:47:38.877424 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877422 4959 cpu_manager.go:414] "containerMap: removing container" podUID="e9ced95e-bb4d-4778-a495-d59783a05932" containerName="revoke-keys" Nov 04 03:47:38.877563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877427 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ced95e-bb4d-4778-a495-d59783a05932" containerName="revoke-keys" Nov 04 03:47:38.877563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877505 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" Nov 04 03:47:38.877563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877514 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df7a391-1ace-4782-8f2c-410d14c4429e" containerName="istio-proxy" Nov 04 03:47:38.877563 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.877523 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ced95e-bb4d-4778-a495-d59783a05932" containerName="revoke-keys" Nov 04 03:47:38.881674 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.881657 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:38.936978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.936935 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 04 03:47:38.937221 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.937201 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 04 03:47:38.937366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.937250 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hlrnm\"" Nov 04 03:47:38.937366 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.937255 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 04 03:47:38.939453 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:38.939428 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5"] Nov 04 03:47:39.003633 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.003598 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.003846 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.003654 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.003846 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.003753 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdjb\" (UniqueName: \"kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.104406 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.104370 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.104623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.104417 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdjb\" (UniqueName: \"kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.104623 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.104510 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.105118 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.105094 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.107028 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.107005 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.164197 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.164114 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:39.166453 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.166436 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdjb\" (UniqueName: \"kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb\") pod \"maas-api-revoke-sub-b0439881-g6qf5\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.195407 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.195379 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:39.416561 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.416457 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:39.566091 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:39.566071 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5"] Nov 04 03:47:39.568302 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:47:39.568269 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c23edb_e12a_430a_b8e8_1ebacb2488a6.slice/crio-6c273b6c6d0a359ab04d1924a502b44329d826b44764965b29379bfc1bba9f49 WatchSource:0}: Error finding container 6c273b6c6d0a359ab04d1924a502b44329d826b44764965b29379bfc1bba9f49: Status 404 returned error can't find the container with id 6c273b6c6d0a359ab04d1924a502b44329d826b44764965b29379bfc1bba9f49 Nov 04 03:47:40.559415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:40.559385 4959 generic.go:356] "Generic (PLEG): container finished" podID="f6c23edb-e12a-430a-b8e8-1ebacb2488a6" containerID="d22aad402b900c3f4e35e7fb07b681e599b4f10c13b218fbff5bd62de7498015" exitCode=0 Nov 04 03:47:40.559778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:40.559485 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" event={"ID":"f6c23edb-e12a-430a-b8e8-1ebacb2488a6","Type":"ContainerDied","Data":"d22aad402b900c3f4e35e7fb07b681e599b4f10c13b218fbff5bd62de7498015"} Nov 04 03:47:40.559778 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:40.559540 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" event={"ID":"f6c23edb-e12a-430a-b8e8-1ebacb2488a6","Type":"ContainerStarted","Data":"6c273b6c6d0a359ab04d1924a502b44329d826b44764965b29379bfc1bba9f49"} Nov 04 03:47:41.708890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.708866 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:41.735144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.735116 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca\") pod \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " Nov 04 03:47:41.735280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.735157 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb\" (UniqueName: \"kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb\") pod \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " Nov 04 03:47:41.735280 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.735214 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca\") pod \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\" (UID: \"f6c23edb-e12a-430a-b8e8-1ebacb2488a6\") " Nov 04 03:47:41.735688 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.735662 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca" pod "f6c23edb-e12a-430a-b8e8-1ebacb2488a6" (UID: "f6c23edb-e12a-430a-b8e8-1ebacb2488a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:47:41.737363 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.737341 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca" pod "f6c23edb-e12a-430a-b8e8-1ebacb2488a6" (UID: "f6c23edb-e12a-430a-b8e8-1ebacb2488a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 04 03:47:41.737363 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.737359 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb" pod "f6c23edb-e12a-430a-b8e8-1ebacb2488a6" (UID: "f6c23edb-e12a-430a-b8e8-1ebacb2488a6"). InnerVolumeSpecName "kube-api-access-mgdjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:41.836357 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.836333 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-xks-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:41.836357 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.836358 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdjb\" (UniqueName: \"kubernetes.io/projected/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-kube-api-access-mgdjb\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:41.836540 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:41.836368 4959 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c23edb-e12a-430a-b8e8-1ebacb2488a6-maas-api-openshift-ca\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:42.571856 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:42.571831 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" Nov 04 03:47:42.572003 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:42.571867 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0439881-g6qf5" event={"ID":"f6c23edb-e12a-430a-b8e8-1ebacb2488a6","Type":"ContainerDied","Data":"6c273b6c6d0a359ab04d1924a502b44329d826b44764965b29379bfc1bba9f49"} Nov 04 03:47:42.572003 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:42.571918 4959 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c273b6c6d0a359ab04d1924a502b44329d826b44764965b29379bfc1bba9f49" Nov 04 03:47:42.619624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:42.619589 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:43.410781 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.410725 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" podUID="94d18bff-f0af-4f31-843c-a01eacce1dee" containerName="payload-processing" containerID="cri-o://fbb094d0f59e5dfd3c4d3703163cae462136e0cbbfd7c7ca8c22d928afba59b0" gracePeriod=25 Nov 04 03:47:43.547260 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.547221 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerName="payload-pre-processing" containerID="cri-o://ba772e25444e56fa59165539b727dc4cea60796373ac8e66f0fdb6fc874e798c" gracePeriod=25 Nov 04 03:47:43.579433 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.579396 4959 generic.go:356] "Generic (PLEG): container finished" podID="94d18bff-f0af-4f31-843c-a01eacce1dee" containerID="fbb094d0f59e5dfd3c4d3703163cae462136e0cbbfd7c7ca8c22d928afba59b0" exitCode=0 Nov 04 03:47:43.579631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.579450 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" event={"ID":"94d18bff-f0af-4f31-843c-a01eacce1dee","Type":"ContainerDied","Data":"fbb094d0f59e5dfd3c4d3703163cae462136e0cbbfd7c7ca8c22d928afba59b0"} Nov 04 03:47:43.671033 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.670976 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:47:43.755230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.755194 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22\" (UniqueName: \"kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22\") pod \"94d18bff-f0af-4f31-843c-a01eacce1dee\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " Nov 04 03:47:43.755425 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.755358 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume\") pod \"94d18bff-f0af-4f31-843c-a01eacce1dee\" (UID: \"94d18bff-f0af-4f31-843c-a01eacce1dee\") " Nov 04 03:47:43.755922 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.755828 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume" pod "94d18bff-f0af-4f31-843c-a01eacce1dee" (UID: "94d18bff-f0af-4f31-843c-a01eacce1dee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:47:43.757590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.757568 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22" pod "94d18bff-f0af-4f31-843c-a01eacce1dee" (UID: "94d18bff-f0af-4f31-843c-a01eacce1dee"). InnerVolumeSpecName "kube-api-access-v4x22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:47:43.856869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.856841 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d18bff-f0af-4f31-843c-a01eacce1dee-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:43.856869 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:43.856868 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4x22\" (UniqueName: \"kubernetes.io/projected/94d18bff-f0af-4f31-843c-a01eacce1dee-kube-api-access-v4x22\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:47:44.586057 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:44.586030 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" Nov 04 03:47:44.586557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:44.586065 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw" event={"ID":"94d18bff-f0af-4f31-843c-a01eacce1dee","Type":"ContainerDied","Data":"15f43e7402a14c16628d34504db09652047da6d8928017a583b7efdbda91d555"} Nov 04 03:47:44.586557 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:44.586124 4959 scope.go:122] "RemoveContainer" containerID="fbb094d0f59e5dfd3c4d3703163cae462136e0cbbfd7c7ca8c22d928afba59b0" Nov 04 03:47:44.758408 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:44.758370 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw"] Nov 04 03:47:44.807615 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:44.807581 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c5bec0-7b4f5976bd-q8ttw"] Nov 04 03:47:45.666073 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:45.666040 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:46.353988 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:47:46.353950 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:47:46.798229 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:46.798154 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d18bff-f0af-4f31-843c-a01eacce1dee" path="/var/lib/kubelet/pods/94d18bff-f0af-4f31-843c-a01eacce1dee/volumes" Nov 04 03:47:48.886635 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:48.886601 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:47:51.398744 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:51.398713 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:51.979821 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:51.979783 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:56.994977 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:56.994939 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:57.299343 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:57.299258 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:47:57.321132 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:57.321108 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-24x8l"] Nov 04 03:47:58.886239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:47:58.886205 4959 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 04 03:48:02.699675 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.699642 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh"] Nov 04 03:48:02.700159 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.700136 4959 cpu_manager.go:414] "containerMap: removing container" podUID="f6c23edb-e12a-430a-b8e8-1ebacb2488a6" containerName="revoke-keys" Nov 04 03:48:02.700246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.700161 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c23edb-e12a-430a-b8e8-1ebacb2488a6" containerName="revoke-keys" Nov 04 03:48:02.700246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.700185 4959 cpu_manager.go:414] "containerMap: removing container" podUID="94d18bff-f0af-4f31-843c-a01eacce1dee" containerName="payload-processing" Nov 04 03:48:02.700246 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.700194 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d18bff-f0af-4f31-843c-a01eacce1dee" containerName="payload-processing" Nov 04 03:48:02.700406 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.700337 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c23edb-e12a-430a-b8e8-1ebacb2488a6" containerName="revoke-keys" Nov 04 03:48:02.700406 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.700349 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d18bff-f0af-4f31-843c-a01eacce1dee" containerName="payload-processing" Nov 04 03:48:02.703559 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.703543 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.750763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.750732 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt"] Nov 04 03:48:02.754972 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.754952 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.771735 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.771712 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c5bec0-openshift-default-dockercfg-fkbcb\"" Nov 04 03:48:02.806958 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.806920 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh"] Nov 04 03:48:02.840733 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840701 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lrdj\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.840733 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840735 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.840930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840752 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxcc\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.840930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840875 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.840930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840903 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.841029 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840959 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.841029 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.840981 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.841029 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841007 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.841124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841073 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841124 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841101 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841168 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841200 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841230 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841215 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841233 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841321 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841280 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.841385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841339 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.841385 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841364 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.841447 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.841413 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.881128 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.881098 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt"] Nov 04 03:48:02.942590 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942558 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.942764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942605 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.942764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942624 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.942764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942667 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lrdj\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.942764 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942684 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942791 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxcc\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942841 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942877 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942914 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942942 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942967 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.943006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.942998 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943033 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943062 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943067 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943125 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943149 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943177 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943228 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.943368 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943266 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.944014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943431 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.944014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943557 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.944014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943639 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.944014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943663 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.944014 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.943854 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.944269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.944075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.944269 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.944075 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.944382 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.944347 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.945381 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.945358 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:02.945624 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.945608 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.945825 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.945809 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:02.946181 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:02.946166 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:03.040268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.040126 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:03.041137 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.041111 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:03.041428 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.041405 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lrdj\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:03.042455 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.042428 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxcc\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc\") pod \"e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:03.068708 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.068668 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:03.313398 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.313312 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:03.316167 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.316148 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:03.433942 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.433911 4959 scope.go:122] "RemoveContainer" containerID="2f6da64edda60036655cd804ea04e50ef87e97a477489309ee3b21441a0048b5" Nov 04 03:48:03.442528 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.442503 4959 scope.go:122] "RemoveContainer" containerID="a4ebd2cc0f33067d071e07305677a1d6ecf7a19807ebef8456f46a5a671aa575" Nov 04 03:48:03.450712 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.450693 4959 scope.go:122] "RemoveContainer" containerID="9831c0faf6a5375a94beb65825d2328cabdb4ffaf4ea5be4003a14e26d881389" Nov 04 03:48:03.458518 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.458497 4959 scope.go:122] "RemoveContainer" containerID="48e252acc98d6aa1e7a36e6dfd8c76340e64ab9f34bf0339a413cc90c70151bf" Nov 04 03:48:03.466048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.466025 4959 scope.go:122] "RemoveContainer" containerID="1fe350488cc83c84801eaa1289ee543ecda8b3e397646d580fcad1306f7d5268" Nov 04 03:48:03.510759 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.510737 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt"] Nov 04 03:48:03.512145 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:48:03.512125 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08693beb_3656_4f56_bfac_b9749af101b0.slice/crio-f300525b8e7ba6b1618332ea6f8afcc21dfd2f70243ce5304c8ae8ce516abdda WatchSource:0}: Error finding container f300525b8e7ba6b1618332ea6f8afcc21dfd2f70243ce5304c8ae8ce516abdda: Status 404 returned error can't find the container with id f300525b8e7ba6b1618332ea6f8afcc21dfd2f70243ce5304c8ae8ce516abdda Nov 04 03:48:03.514063 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.514036 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:48:03.514133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.514098 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:48:03.514170 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.514144 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:48:03.558887 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.558861 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:03.683742 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.683692 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" event={"ID":"08693beb-3656-4f56-bfac-b9749af101b0","Type":"ContainerStarted","Data":"47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192"} Nov 04 03:48:03.683913 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.683767 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" event={"ID":"08693beb-3656-4f56-bfac-b9749af101b0","Type":"ContainerStarted","Data":"f300525b8e7ba6b1618332ea6f8afcc21dfd2f70243ce5304c8ae8ce516abdda"} Nov 04 03:48:03.685726 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.685693 4959 generic.go:356] "Generic (PLEG): container finished" podID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerID="ba772e25444e56fa59165539b727dc4cea60796373ac8e66f0fdb6fc874e798c" exitCode=0 Nov 04 03:48:03.685876 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.685760 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" event={"ID":"9ef40689-e3ac-41df-a040-5bc335bce2d2","Type":"ContainerDied","Data":"ba772e25444e56fa59165539b727dc4cea60796373ac8e66f0fdb6fc874e798c"} Nov 04 03:48:03.696746 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.695658 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh"] Nov 04 03:48:03.699207 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:48:03.697447 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbb5634_70c2_443d_b9bd_6435817f310a.slice/crio-2b97c0c2ea77ab9c054e91bef4199438e15dab935c124cf01d4747dfb9cb9950 WatchSource:0}: Error finding container 2b97c0c2ea77ab9c054e91bef4199438e15dab935c124cf01d4747dfb9cb9950: Status 404 returned error can't find the container with id 2b97c0c2ea77ab9c054e91bef4199438e15dab935c124cf01d4747dfb9cb9950 Nov 04 03:48:03.700596 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.700566 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:48:03.700932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.700660 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:48:03.700932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.700701 4959 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Nov 04 03:48:03.801420 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.801394 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:48:03.806988 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.806936 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podStartSLOduration=1.80691897 podStartE2EDuration="1.80691897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:03.805372363 +0000 UTC m=+1861.188800404" watchObservedRunningTime="2026-11-04 03:48:03.80691897 +0000 UTC m=+1861.190347074" Nov 04 03:48:03.953582 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.953539 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume\") pod \"9ef40689-e3ac-41df-a040-5bc335bce2d2\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " Nov 04 03:48:03.953776 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.953585 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld\" (UniqueName: \"kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld\") pod \"9ef40689-e3ac-41df-a040-5bc335bce2d2\" (UID: \"9ef40689-e3ac-41df-a040-5bc335bce2d2\") " Nov 04 03:48:03.954017 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.953992 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume" pod "9ef40689-e3ac-41df-a040-5bc335bce2d2" (UID: "9ef40689-e3ac-41df-a040-5bc335bce2d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:48:03.960505 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:03.956526 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld" pod "9ef40689-e3ac-41df-a040-5bc335bce2d2" (UID: "9ef40689-e3ac-41df-a040-5bc335bce2d2"). InnerVolumeSpecName "kube-api-access-kb7ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:04.057504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.055323 4959 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ef40689-e3ac-41df-a040-5bc335bce2d2-plugins-config-volume\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:04.057504 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.055367 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb7ld\" (UniqueName: \"kubernetes.io/projected/9ef40689-e3ac-41df-a040-5bc335bce2d2-kube-api-access-kb7ld\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:04.069728 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.069686 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:04.071088 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.071060 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 04 03:48:04.071219 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.071111 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 04 03:48:04.728495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.727125 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" event={"ID":"9ef40689-e3ac-41df-a040-5bc335bce2d2","Type":"ContainerDied","Data":"bdc9799091cf3d47c77181b31fe6ca6c9e108ce3733738b3306ddd5bc2b5145c"} Nov 04 03:48:04.728495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.727190 4959 scope.go:122] "RemoveContainer" containerID="ba772e25444e56fa59165539b727dc4cea60796373ac8e66f0fdb6fc874e798c" Nov 04 03:48:04.728495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.727308 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk" Nov 04 03:48:04.734415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.732996 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" event={"ID":"7cbb5634-70c2-443d-b9bd-6435817f310a","Type":"ContainerStarted","Data":"da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089"} Nov 04 03:48:04.734415 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.733049 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" event={"ID":"7cbb5634-70c2-443d-b9bd-6435817f310a","Type":"ContainerStarted","Data":"2b97c0c2ea77ab9c054e91bef4199438e15dab935c124cf01d4747dfb9cb9950"} Nov 04 03:48:04.967184 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:04.967133 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podStartSLOduration=2.9671198519999997 podStartE2EDuration="2.967119852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-04 03:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:48:04.966226084 +0000 UTC m=+1862.349654123" watchObservedRunningTime="2026-11-04 03:48:04.967119852 +0000 UTC m=+1862.350547894" Nov 04 03:48:05.069282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.069198 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 04 03:48:05.069282 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.069253 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 04 03:48:05.103268 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.103222 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk"] Nov 04 03:48:05.113305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.113254 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c5bec0-ddbf5d8ff-mr6wk"] Nov 04 03:48:05.318495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.318143 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:05.318495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.318267 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 03:48:05.318495 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:05.318310 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 03:48:06.069774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:06.069521 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 04 03:48:06.069774 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:06.069589 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 04 03:48:06.318496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:06.317685 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 03:48:06.318496 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:06.317747 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 03:48:06.832384 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:06.830231 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" path="/var/lib/kubelet/pods/9ef40689-e3ac-41df-a040-5bc335bce2d2/volumes" Nov 04 03:48:07.069925 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:07.069891 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 04 03:48:07.070294 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:07.069942 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 04 03:48:07.326551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:07.316936 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 03:48:07.326551 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:07.316998 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 03:48:08.069306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.069274 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 04 03:48:08.069571 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.069546 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 04 03:48:08.321026 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.320932 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 03:48:08.321625 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.321580 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 03:48:08.353892 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.353865 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:08.615494 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.611098 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:08.858502 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:08.858447 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:09.069820 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:09.069743 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 04 03:48:09.070025 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:09.070001 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 04 03:48:09.325556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:09.323953 4959 patch_prober.go:29] interesting pod/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 04 03:48:09.325556 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:09.324034 4959 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 04 03:48:10.078414 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:10.078371 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:10.079320 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:10.079297 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:10.082006 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:10.081981 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:10.321302 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:10.321273 4959 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:10.321541 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:10.321515 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:10.322136 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:10.322118 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:11.655482 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:11.655426 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:12.051598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:12.051524 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh"] Nov 04 03:48:12.065092 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:12.065061 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt"] Nov 04 03:48:12.159974 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:12.159933 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:12.782383 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:12.782318 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" containerID="cri-o://47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192" gracePeriod=30 Nov 04 03:48:12.782383 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:12.782355 4959 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" containerID="cri-o://da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089" gracePeriod=30 Nov 04 03:48:14.408707 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:14.408670 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:14.666645 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:14.666553 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:15.565066 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:15.565041 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-24x8l_eab5b6b7-8940-4e9e-91d8-d3679f390a34/limitador/0.log" Nov 04 03:48:16.355565 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:48:16.355532 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:48:17.244041 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:17.244008 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zt9qt_6f9e914b-c63a-409b-9b2f-0c494ef2620b/manager/0.log" Nov 04 03:48:18.041829 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.041805 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:18.068158 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.068136 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:18.156525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156490 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.156525 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156529 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.156763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156554 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.156763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156618 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.156763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156652 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.156763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156682 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.156763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156716 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.156763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156748 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156779 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156809 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156843 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156867 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156892 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156914 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.156929 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157003 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157004 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157043 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy\") pod \"08693beb-3656-4f56-bfac-b9749af101b0\" (UID: \"08693beb-3656-4f56-bfac-b9749af101b0\") " Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157078 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157108 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157144 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157100 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157151 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157206 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157230 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157248 4959 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo\") pod \"7cbb5634-70c2-443d-b9bd-6435817f310a\" (UID: \"7cbb5634-70c2-443d-b9bd-6435817f310a\") " Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157365 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157374 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157670 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157689 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb5634-70c2-443d-b9bd-6435817f310a-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157706 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157721 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157734 4959 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-credential-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157748 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157754 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157761 4959 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-socket\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157797 4959 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08693beb-3656-4f56-bfac-b9749af101b0-istiod-ca-cert\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.157882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.157815 4959 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-data\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.159868 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.159841 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:18.160053 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.160027 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:48:18.160171 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.160156 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "kube-api-access-8lrdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:18.160404 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.160372 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:18.160404 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.160378 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.160606 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.160592 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 04 03:48:18.160677 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.160639 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo" pod "7cbb5634-70c2-443d-b9bd-6435817f310a" (UID: "7cbb5634-70c2-443d-b9bd-6435817f310a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 04 03:48:18.161181 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.161159 4959 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc" pod "08693beb-3656-4f56-bfac-b9749af101b0" (UID: "08693beb-3656-4f56-bfac-b9749af101b0"). InnerVolumeSpecName "kube-api-access-bgxcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 04 03:48:18.206519 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.206496 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:18.259323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259293 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08693beb-3656-4f56-bfac-b9749af101b0-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259323 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259325 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxcc\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-kube-api-access-bgxcc\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259342 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259356 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259369 4959 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb5634-70c2-443d-b9bd-6435817f310a-workload-certs\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259394 4959 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08693beb-3656-4f56-bfac-b9749af101b0-istio-envoy\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259407 4959 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lrdj\" (UniqueName: \"kubernetes.io/projected/7cbb5634-70c2-443d-b9bd-6435817f310a-kube-api-access-8lrdj\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259422 4959 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb5634-70c2-443d-b9bd-6435817f310a-istio-podinfo\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.259754 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.259436 4959 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08693beb-3656-4f56-bfac-b9749af101b0-istio-token\") on node \"ip-10-0-132-184.us-west-1.compute.internal\" DevicePath \"\"" Nov 04 03:48:18.708444 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.708412 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm64c_8bfb06a1-e89e-4063-bf27-e580ae21a09b/manager/0.log" Nov 04 03:48:18.816267 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.816234 4959 generic.go:356] "Generic (PLEG): container finished" podID="08693beb-3656-4f56-bfac-b9749af101b0" containerID="47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192" exitCode=0 Nov 04 03:48:18.816419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.816297 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" Nov 04 03:48:18.816419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.816323 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" event={"ID":"08693beb-3656-4f56-bfac-b9749af101b0","Type":"ContainerDied","Data":"47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192"} Nov 04 03:48:18.816419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.816393 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt" event={"ID":"08693beb-3656-4f56-bfac-b9749af101b0","Type":"ContainerDied","Data":"f300525b8e7ba6b1618332ea6f8afcc21dfd2f70243ce5304c8ae8ce516abdda"} Nov 04 03:48:18.816601 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.816421 4959 scope.go:122] "RemoveContainer" containerID="47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192" Nov 04 03:48:18.818012 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.817967 4959 generic.go:356] "Generic (PLEG): container finished" podID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerID="da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089" exitCode=0 Nov 04 03:48:18.818132 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.818054 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" event={"ID":"7cbb5634-70c2-443d-b9bd-6435817f310a","Type":"ContainerDied","Data":"da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089"} Nov 04 03:48:18.818132 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.818094 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" event={"ID":"7cbb5634-70c2-443d-b9bd-6435817f310a","Type":"ContainerDied","Data":"2b97c0c2ea77ab9c054e91bef4199438e15dab935c124cf01d4747dfb9cb9950"} Nov 04 03:48:18.818307 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.818170 4959 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh" Nov 04 03:48:18.827256 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.827241 4959 scope.go:122] "RemoveContainer" containerID="47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192" Nov 04 03:48:18.827576 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:48:18.827553 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192\": container with ID starting with 47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192 not found: ID does not exist" containerID="47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192" Nov 04 03:48:18.827631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.827589 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192"} err="failed to get container status \"47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192\": rpc error: code = NotFound desc = could not find container \"47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192\": container with ID starting with 47de8cdc32661709ae1774de780da00aa7a5d3e268896dd8d63348595c101192 not found: ID does not exist" Nov 04 03:48:18.827631 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.827606 4959 scope.go:122] "RemoveContainer" containerID="da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089" Nov 04 03:48:18.835663 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.835648 4959 scope.go:122] "RemoveContainer" containerID="da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089" Nov 04 03:48:18.835914 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:48:18.835895 4959 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089\": container with ID starting with da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089 not found: ID does not exist" containerID="da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089" Nov 04 03:48:18.835961 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:18.835924 4959 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089"} err="failed to get container status \"da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089\": rpc error: code = NotFound desc = could not find container \"da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089\": container with ID starting with da4a99965e58eb3920b95a7f893b5a3d43a32d962dcc64432256f2a27173a089 not found: ID does not exist" Nov 04 03:48:19.033457 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:19.033378 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh"] Nov 04 03:48:19.083930 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:19.083899 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-z2kzh"] Nov 04 03:48:19.148881 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:19.148853 4959 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt"] Nov 04 03:48:19.155435 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:19.155398 4959 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c5bec0-openshift-default-6f85d9855f-j2rjt"] Nov 04 03:48:20.798388 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:20.798353 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08693beb-3656-4f56-bfac-b9749af101b0" path="/var/lib/kubelet/pods/08693beb-3656-4f56-bfac-b9749af101b0/volumes" Nov 04 03:48:20.798890 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:20.798826 4959 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" path="/var/lib/kubelet/pods/7cbb5634-70c2-443d-b9bd-6435817f310a/volumes" Nov 04 03:48:33.948932 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:33.948888 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:48:46.354190 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:48:46.354147 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:48:55.139225 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:55.139195 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zt9qt_6f9e914b-c63a-409b-9b2f-0c494ef2620b/manager/0.log" Nov 04 03:48:56.778142 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:48:56.778105 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm64c_8bfb06a1-e89e-4063-bf27-e580ae21a09b/manager/0.log" Nov 04 03:49:03.659417 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:03.659380 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:49:08.418685 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:08.418650 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-35d81421-sg6vn_1e6e51a3-6144-4a00-be33-eb9c2272e81a/revoke-keys/0.log" Nov 04 03:49:08.653962 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:08.653933 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7442a4c3-fn55c_e9ced95e-bb4d-4778-a495-d59783a05932/revoke-keys/0.log" Nov 04 03:49:08.868482 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:08.868451 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b0439881-g6qf5_f6c23edb-e12a-430a-b8e8-1ebacb2488a6/revoke-keys/0.log" Nov 04 03:49:13.086678 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:13.086604 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-24x8l_eab5b6b7-8940-4e9e-91d8-d3679f390a34/limitador/0.log" Nov 04 03:49:14.725909 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:14.725878 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7jh5t_d9146acf-ce25-4bff-b1f3-04b5b112052a/discovery/0.log" Nov 04 03:49:15.747730 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:15.747698 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-5s8q9_d86f1fc3-dd20-4598-a5d0-03c40cbc15c5/istio-proxy/0.log" Nov 04 03:49:16.022008 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:16.021924 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-6s4br_6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7/istio-proxy/0.log" Nov 04 03:49:16.354129 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:49:16.354100 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:49:16.509156 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:16.509124 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-hj5bm_7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd/istio-proxy/0.log" Nov 04 03:49:16.751849 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:16.751773 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-hx8xk_c1ec031a-04b7-4479-a109-0c1749bc2fe9/istio-proxy/0.log" Nov 04 03:49:28.637418 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:28.637390 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66bfccb89c-84k2x_6802ea74-e881-496d-b63b-601d10ae7d02/router/0.log" Nov 04 03:49:30.446802 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:30.446773 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-pqqxp_dbf9e67f-87bf-43da-a257-d8a2d6171613/main/0.log" Nov 04 03:49:30.950111 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:30.950080 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-zg7ps_0057f972-ac5a-420a-856d-f5cbac9d7622/main/0.log" Nov 04 03:49:31.451915 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:31.451887 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-x4fz5_72981a99-2a4a-4b23-aaf3-84733ccb53b5/main/0.log" Nov 04 03:49:46.354940 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:49:46.354906 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:49:51.892007 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:51.891971 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:49:52.191780 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:49:52.191684 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:50:01.677682 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:01.677648 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cjhm7_3d6a461d-3472-4285-b5a9-1a11367f6497/global-pull-secret-syncer/0.log" Nov 04 03:50:02.896239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:02.896213 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7n6xm_f762673f-00d6-4182-b7e3-bd16847fd218/connection-checker/0.log" Nov 04 03:50:03.366048 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:03.366023 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-g5nm2_781b84f6-581f-4357-b447-8d4f92aece81/connection-checker/0.log" Nov 04 03:50:03.787306 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:03.787228 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-n2zsd_a699a906-fae8-47d9-92ba-8eefce6cf724/connection-checker/0.log" Nov 04 03:50:04.011057 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:04.011028 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:50:04.285239 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:04.285212 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" Nov 04 03:50:05.265076 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:05.265046 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zt9qt_6f9e914b-c63a-409b-9b2f-0c494ef2620b/manager/0.log" Nov 04 03:50:05.675133 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:05.675097 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-184.us-west-1.compute.internal_7cf4b9ea98890a60ae37fd39b8c09d27/haproxy/0.log" Nov 04 03:50:07.142345 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:07.142305 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm64c_8bfb06a1-e89e-4063-bf27-e580ae21a09b/manager/0.log" Nov 04 03:50:11.111792 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:11.111736 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zt9qt_6f9e914b-c63a-409b-9b2f-0c494ef2620b/manager/0.log" Nov 04 03:50:12.613196 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613158 4959 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst"] Nov 04 03:50:12.613882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613742 4959 cpu_manager.go:414] "containerMap: removing container" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerName="payload-pre-processing" Nov 04 03:50:12.613882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613760 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerName="payload-pre-processing" Nov 04 03:50:12.613882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613788 4959 cpu_manager.go:414] "containerMap: removing container" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" Nov 04 03:50:12.613882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613796 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" Nov 04 03:50:12.613882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613820 4959 cpu_manager.go:414] "containerMap: removing container" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" Nov 04 03:50:12.613882 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613825 4959 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" Nov 04 03:50:12.614130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613909 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cbb5634-70c2-443d-b9bd-6435817f310a" containerName="istio-proxy" Nov 04 03:50:12.614130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613927 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef40689-e3ac-41df-a040-5bc335bce2d2" containerName="payload-pre-processing" Nov 04 03:50:12.614130 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.613939 4959 memory_manager.go:365] "RemoveStaleState removing state" podUID="08693beb-3656-4f56-bfac-b9749af101b0" containerName="istio-proxy" Nov 04 03:50:12.617400 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.617377 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.725375 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.725343 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnkxc\"/\"kube-root-ca.crt\"" Nov 04 03:50:12.725598 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.725542 4959 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rnkxc\"/\"openshift-service-ca.crt\"" Nov 04 03:50:12.727689 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.727652 4959 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rnkxc\"/\"default-dockercfg-hbjsz\"" Nov 04 03:50:12.729224 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.729197 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst"] Nov 04 03:50:12.815305 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.815265 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-proc\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.815524 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.815338 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-sys\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.815524 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.815401 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrwm\" (UniqueName: \"kubernetes.io/projected/2c764e9e-edfe-41af-88ed-40ca721b27d8-kube-api-access-zjrwm\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.815524 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.815497 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-lib-modules\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.815739 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.815545 4959 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-podres\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916049 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrwm\" (UniqueName: \"kubernetes.io/projected/2c764e9e-edfe-41af-88ed-40ca721b27d8-kube-api-access-zjrwm\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916143 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916139 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-lib-modules\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916178 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-podres\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916224 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-proc\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916264 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-lib-modules\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916277 4959 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-sys\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916325 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-proc\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916377 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916364 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-sys\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.916727 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.916374 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c764e9e-edfe-41af-88ed-40ca721b27d8-podres\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:12.979509 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:12.979440 4959 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrwm\" (UniqueName: \"kubernetes.io/projected/2c764e9e-edfe-41af-88ed-40ca721b27d8-kube-api-access-zjrwm\") pod \"perf-node-gather-daemonset-r6tst\" (UID: \"2c764e9e-edfe-41af-88ed-40ca721b27d8\") " pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:13.011832 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:13.011801 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm64c_8bfb06a1-e89e-4063-bf27-e580ae21a09b/manager/0.log" Nov 04 03:50:13.233493 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:13.233384 4959 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:13.605126 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:13.605105 4959 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst"] Nov 04 03:50:13.607603 ip-10-0-132-184 kubenswrapper[4959]: W1104 03:50:13.607577 4959 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2c764e9e_edfe_41af_88ed_40ca721b27d8.slice/crio-d6cadb1ed3e9b8c445fb3d2d8dbe907160f0311fc37fca170568168bb15d5389 WatchSource:0}: Error finding container d6cadb1ed3e9b8c445fb3d2d8dbe907160f0311fc37fca170568168bb15d5389: Status 404 returned error can't find the container with id d6cadb1ed3e9b8c445fb3d2d8dbe907160f0311fc37fca170568168bb15d5389 Nov 04 03:50:13.616030 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:13.616008 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-24x8l_eab5b6b7-8940-4e9e-91d8-d3679f390a34/limitador/0.log" Nov 04 03:50:14.364245 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:14.364195 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" event={"ID":"2c764e9e-edfe-41af-88ed-40ca721b27d8","Type":"ContainerStarted","Data":"fa25f60a3e00c1faeb036422abbdcd032ab86f10f2bcd3fb21a62b639b7d9c19"} Nov 04 03:50:14.364419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:14.364259 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:14.364419 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:14.364283 4959 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" event={"ID":"2c764e9e-edfe-41af-88ed-40ca721b27d8","Type":"ContainerStarted","Data":"d6cadb1ed3e9b8c445fb3d2d8dbe907160f0311fc37fca170568168bb15d5389"} Nov 04 03:50:14.481906 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:14.481858 4959 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" podStartSLOduration=2.481843722 podStartE2EDuration="2.481843722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-04 03:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-04 03:50:14.481443978 +0000 UTC m=+1991.864872019" watchObservedRunningTime="2026-11-04 03:50:14.481843722 +0000 UTC m=+1991.865271802" Nov 04 03:50:16.354384 ip-10-0-132-184 kubenswrapper[4959]: E1104 03:50:16.354348 4959 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c44w7" containerName="extractor" Nov 04 03:50:16.389542 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:16.389520 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zt9qt_6f9e914b-c63a-409b-9b2f-0c494ef2620b/manager/0.log" Nov 04 03:50:18.254724 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:18.254689 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zm64c_8bfb06a1-e89e-4063-bf27-e580ae21a09b/manager/0.log" Nov 04 03:50:19.647832 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:19.647799 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xlwwg_986942d0-73ec-4567-9026-48482bd186ae/cluster-monitoring-operator/0.log" Nov 04 03:50:20.387118 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:20.387094 4959 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rnkxc/perf-node-gather-daemonset-r6tst" Nov 04 03:50:23.371951 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:23.371921 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7jh5t_d9146acf-ce25-4bff-b1f3-04b5b112052a/discovery/0.log" Nov 04 03:50:24.140083 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:24.140048 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-5s8q9_d86f1fc3-dd20-4598-a5d0-03c40cbc15c5/istio-proxy/0.log" Nov 04 03:50:24.364137 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:24.364109 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-6s4br_6e9eae8c-b48a-46b0-afcf-70f1dec0c1d7/istio-proxy/0.log" Nov 04 03:50:24.827315 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:24.827283 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-hj5bm_7f6a2e5a-d8b8-4c77-a9dc-d55f318f58dd/istio-proxy/0.log" Nov 04 03:50:24.900508 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:24.900459 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhbb6_06463d76-3838-4853-aab5-ae5c5ed6e275/node-exporter/0.log" Nov 04 03:50:25.043638 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:25.043607 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-hx8xk_c1ec031a-04b7-4479-a109-0c1749bc2fe9/istio-proxy/0.log" Nov 04 03:50:25.317619 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:25.317533 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhbb6_06463d76-3838-4853-aab5-ae5c5ed6e275/kube-rbac-proxy/0.log" Nov 04 03:50:25.743870 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:25.743843 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhbb6_06463d76-3838-4853-aab5-ae5c5ed6e275/init-textfile/0.log" Nov 04 03:50:26.767763 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:26.767733 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66bfccb89c-84k2x_6802ea74-e881-496d-b63b-601d10ae7d02/router/0.log" Nov 04 03:50:28.422978 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:28.422715 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:50:28.675221 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:28.675128 4959 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5xngn_dab481c8-cff5-4c29-aa52-552434b043bf/konnectivity-agent/0.log" limit=1024 Nov 04 03:50:31.970043 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:31.970004 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-24x8l_eab5b6b7-8940-4e9e-91d8-d3679f390a34/limitador/0.log" Nov 04 03:50:36.977648 ip-10-0-132-184 kubenswrapper[4959]: I1104 03:50:36.977618 4959 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7jh5t_d9146acf-ce25-4bff-b1f3-04b5b112052a/discovery/0.log"